Archive

The log files processed by EventLog Analyzer are archived periodically for internal, forensic, and compliance audits. You can configure the following as per your requirements:

• Archiving interval
• Type of logs that need to be archived
• Storage location of the archived files
• Retention period

The archived files can be encrypted and time-stamped to make them secure and tamper-proof.

How to view archived logs ?

To view your archived log data, go to the Settings tab in EventLog Analyzer and navigate to Admin Settings > Data Storage > Archives

The Archived Logs page contains the following information:

• Device - List of devices from which the logs are being collected
• Format - Device type
• From and To - The time frame denotes the time period during which the logs were collected and archived by EventLog Analyzer.
• Size - Size of the archived log data collected from each device.
• Integrity - The integrity of the archived files, whether they are intact or have been tampered with, is denoted by the following states:
1. Verified - Archived logs are intact.
2. Archive file is missing - When the flat file is not found during the compression/zipping process.
3. Archive file not found - When an archived file is not available in the location where it was originally stored in the DB.
4. Archive file is tampered - When the original archive file is edited/some part of the file is deleted externally.

Note: In case a file has been deleted or tampered with, an email notification will be sent immediately containing the message "Archive file is tampered".

5. Archive file available - When the archive integrity check is disabled, both the verified and tampered files will carry this status.
6. Archive file not available - When the archive integrity check is disabled and the archive file is either missing or not found in the original location, this status will be shown.
• The status of the archival is indicated by the following four different states:
1. Loaded - The archived files are already loaded to the database. Click View to view the file
2. Data already available - If the archive file is in Elastic Search database
3. Data partially available - If some of the archive data is in ElasticSearch database
4. Not Loaded - If the archive file is not in ElasticSearch database.

How to view a specific archival file?

• To view a specific archival file, click on the check box corresponding to Device.
• To view the log files that were archived during a specific time, click on the calendar icon in the top right corner of the page and select the desired period.

How to filter and view a set of archive files?

To view files based on the size or status of the archive data, click on the filter icon next to Size or Status and set the appropriate values. The files will be filtered based on the given values.

How to sort the list of archive files?

Click on the drop down icon next to Device/From/To, to sort the list in ascending order based on the respective column values. By clicking again, the list will be sorted in descending order.

How to load archive files?

To load your archived files, go to the Settings page in EventLog Analyzer and navigate to Admin Settings > Data Storage > Archives

1. Check the status of the archived file corresponding to the device. If it shows Not Loaded, click on the Load Archive button to load the files to the database.
2. Once the status of the file changes to Loaded, click on the corresponding View button to view the files.

Note: To unload a file, select the file and click on the Unload Archive button.

Note: If the status of the file says Data partially available and if you proceed to load the archive, there could be a duplication of the data.

How to delete archive files ?

To delete your archived files, go to the Settings page in EventLog Analyzer and navigate to Admin Settings > Data Storage > Archives.

1. Select the archived file(s) by selecting the respective check box(es).
2. Delete the archived file(s) by clicking on the Delete icon.

How to configure group based/device based archive settings ?

To configure archival settings, click on Settings in the top right corner of the screen.

Configure the archive interval, retention period, encryption, time-stamp of the archive files, location to save the archive files and the index files.

Note: The archive and database storage are asynchronus operations. These operations are unrelated.

1. Ensure that archiving is enabled. By default, it is enabled.Use the toggle button to disable archiving.
2. To secure the archive files, enable encryption. By default, it will be disabled.
3. Enter the Archive retention period for the archived files. The default period is forever.
4. Logs can be archived in two formats - Raw Logs with Parsed Fields and Raw Logs. Logs will be stored with metadata on selecting the former, and without metadata for the latter.

Note: The storage space for Raw Logs will be lesser but only basic reports can be generated using this data.

5. Enter the storage location for the archived files in the Archive Location field. Click on Verify Location to validate the location.
6. Enter the log retention period for the loaded archive files. The default period is 7 days.
7. Click on Advanced and fill in the following fields:
1. Choose the time interval for file creation. The logs will be written to flat files at the specified time period.

Note: The default interval is 8 hours.

2. Choose the required time interval for creating a zip file. The flat files will be compressed (40:1 ratio) and zip files are created at the specified time period.

Note: The default interval is 1 day.

3. Enable Archive Timestamping if required. By default, it is disabled.
4. The Periodic Archive Integrity Check is enabled by default.

Note: The default interval is 1 day.

8. Save the settings and close the window. For instant archiving, click the Zip Now button next to Zip Creational Interval.

Configure multiple archive settings by clicking on Create New Policy in the top right corner.

Additional configuration - Select the devices/groups for which the policy will be applied.

How to view configured Policy ?

Click on Settings at the top right corner of the screen. This will lead to the Archive Settings page which contains all the configured policies.

• Policy Name - Specifies the name of the policy.
• Archive Location - Shows the location of the policy.
• Devices/Groups - Shows all the devices and groups added in the policy.
• Size - Total size of archive of all the devices/groups added in the policy.
• Retention period - Log retention period of the policy.
• Status - Shows the status of the archival. The status will either be Success or Archiving Disabled.

Click on Edit by hovering on the policy to edit the configured settings.

You can also add a new policy by clicking on the Create New Policy button in the top right corner in archive settings page.

How to edit the priority of the policies?

To change the priority of the policies, click on Priority Policy, rearrange the policies by dragging and dropping them, and save.

Note: If a device/group has been added under multiple policies, the archive settings of the policy with the highest priority will be applied to that particular device/group.

How to check to which policy applies to a specific device?

In the Settings tab of EventLog Analyzer, navigate to Admin Settings > Data Storage > Archives > Settings > Archive Summary

• Device - Shows the list of devices that are added in one or more policies
• Effective Policy Applied - Shows the policy which is applied to that particular device.
• Location - Shows the location of the policy.
• Total size - Shows the total size of archives for that particular device.
• Size in location - Shows the size of the device archives collected under that specific policy.

Archive troubleshooting cases :

1. Update path
• Goto Settings > Admin Settings > Data Storage > Archives > More in the top right corner > Update path
• Select the old archive location in dropdown and enter the new location where archives are moved or present in Archives moved location and click on Update.

2. Update archive file integrity
• Goto Settings > Admin Settings > Data Storage > Archives > More > Update path.
• Click the refresh button in the top right corner to update the integrity status of the files.

The File not Found status will change to Verified, if the file is present in the directory as specified in DB. This will also change the status from Tampered Files to Verified.


3. To add archives in DB
• Goto Settings > Admin Settings > Data Storage > Archives > More > Add Archive Entries.
• Enter the location where the archives are present.If needed, select Device and add the archives of a particular device.

4. If ES/data lost or corrupted
• Goto Settings > Admin Settings > Data Storage > Archives > More > Rebuild Indexes.
• Select the date range and the device for which the logs need to be indexed in ES from Archives.Click on Rebuild.

Steps to move EventLog Analyzer's Elasticsearch indices to a new location

Case 1: EventLog Analyzer as a standalone setup (Not integrated with Log360)

1. Shutdown EventLog Analyzer.
2. Navigate to <Eventlog home>\ES\config\elasticsearch.yml, update path.data to include the new location and save the file.
3. Move the files from <ManageEngine>\<Eventlog>\ES\data folder to the new location.

Case 2: EventLog Analyzer is integrated into Log360 and is installed with Log360 installer (Bundled):

In this case, EventLog Analyzer uses a common ES that's shared with other modules

Note: With Log360, the integrated module will have only one ES and it can be located in the Admin > Administration and Search Engine Management page. By clicking on details we can see that it is running from <ManageEngine>\elasticsearch\ES folder.

1. Shutdown EventLog Analyzer and Log360.
2. Shutdown common ES.
1. Open Command Prompt as the Administrator in <ManageEngine>\elasticsearch\ES\bin
2. Run stopES.bat
3. Navigate to <ManageEngine>\elasticsearch\ES\config\elasticsearch.yml, update path.data to include the new location and save the file.
4. Move the files from <ManageEngine>\elasticsearch\ES\data folder to the new location.

Case 3: EventLog Analyzer is manually integrated into Log360:

In this case, EventLog Analyzer will be using its existing (before integration) local and the common ES (after integration with Log360).

Note:By default, the integrated module will have two ES and it can be located in the Admin > Administration and Search Engine Management page. By clicking on details we can see that one is running from EventLog Analyzer <Eventlog home>\ES folder and other from <ManageEngine>\elasticsearch\ES folder.

1. Shutdown EventLog Analyzer and Log360.
2. Shutdown common ES.
1. Open Command Prompt as the Administrator in <ManageEngine>\elasticsearch\ES\bin
2. Run stopES.bat
3. Navigate to <ManageEngine>\elasticsearch\ES\config\elasticsearch.yml, update path.data to include the new location and save the file.
4. Move the files from <ManageEngine>\elasticsearch\ES\data folder to the new location.
5. Navigate to <ManageEngine>\<Eventlog>\ES\config\elasticsearch.yml, update path.data to include the new location (different from the one given for common ES) and save the file.
6. Move the files from <ManageEngine>\<Eventlog>\ES\data folder to the new location.

Steps to move EventLog Analyzer's Elasticsearch data to a new location

Case 1: EventLog Analyzer as a standalone setup (Not integrated with Log360)

1. Shutdown EventLog Analyzer.
2. Navigate to <Eventlog home>\ES\config\elasticsearch.yml, update path.data to include the new data location and save the file.
3. In <Eventlog home>\ES\config\elasticsearch.yml, update path.repo to include the new repository location (parallel to data directory) and save the file.
4. Move the files from <ManageEngine>\<Eventlog>\ES\data folder to the new location.
5. Create a folder with the name archive (parallel to the new data directory).
6. Move the files from <ManageEngine>\<Eventlog>\ES\archive folder to the new folder named archive.

Case 2: EventLog Analyzer is integrated into Log360 and is installed with Log360 installer (Bundled):

In this case, EventLog Analyzer uses a common ES that's shared with other modules

Note: With Log360, the integrated module will have only one ES and it can be located in the Admin > Administration and Search Engine Management page. By clicking on details we can see that it is running from <ManageEngine>\elasticsearch\ES folder.

1. Shutdown EventLog Analyzer and Log360.
2. Shutdown common ES.
1. Open Command Prompt as the Administrator in <ManageEngine>\elasticsearch\ES\bin
2. Run stopES.bat
3. Navigate to <ManageEngine>\elasticsearch\ES\config\elasticsearch.yml, update path.data to include the new data location and save the file.
4. Also update path.data in <Eventlog home>\ES\config\elasticsearch.yml to include the new data location (same data location as mentioned in step 3).
5. Update path.repo in <ManageEngine>\elasticsearch\ES\config\elasticsearch.yml to the new repository location (parallel to the new data path).
6. Update path.repo in <Eventlog home>\ES\config\elasticsearch.yml to the new repository location (same repository location as mentioned in step 5).
7. Move the files from <ManageEngine>\elasticsearch\ES\data to the new location.
8. Create a folder with the name archive (parallel to the new data directory).
9. Move the files from <ManageEngine>\<Eventlog>\ES\archive folder to the new folder named archive.

Case 3: EventLog Analyzer is manually integrated into Log360:

In this case, EventLog Analyzer will be using its existing (before integration) local and the common ES (after integration with Log360).

Note:By default, the integrated module will have two ES and it can be located in the Admin > Administration and Search Engine Management page. By clicking on details we can see that one is running from EventLog Analyzer, <Eventlog home>\ES folder and the other from <ManageEngine>\elasticsearch\ES folder.

1. Shutdown EventLog Analyzer and Log360.
2. Shutdown common ES.
3. Open Command Prompt as the Administrator in <ManageEngine>\elasticsearch\ES\bin
4. Run stopES.bat

I. Change in common ES

1. Navigate to <ManageEngine>\elasticsearch\ES\config\elasticsearch.yml, update path.data to include the new location and save the file.
2. Update path.repo in <ManageEngine>\elasticsearch\ES\config\elasticsearch.yml to include the new repository location (parallel to path.data).
3. Move the files from <ManageEngine>\elasticsearch\ES\data to the new location.

II. Change in local ES (the path here should be different from the one given for common ES)

1. Navigate to <ManageEngine>\<Eventlog>\ES\config\elasticsearch.yml, update path.data to include the new location (this should be different from the one given for common ES) and save the file.
2. Update path.repo in <ManageEngine>\<Eventlog home>\ES\config\elasticsearch.yml to the same repository location as that of common ES.
3. Create a folder with the name archive (parallel to the new data directory).
4. Move the files from <ManageEngine>\<Eventlog>\ES\data to the new location.
5. Move the files from <ManageEngine>\<Eventlog>\ES\archive folder to the new folder named archive.