Ongoing problem confirmed by other users.
At 2019-02-03 14:27 it deleted the OP code and Registered Owner of 4AAA12.
Since 2017, 3 out of 7 helicopters from the same owner (Swedish Police) has had the Reg Owner reset to something coming from the main database.
Please try to find the problem.
/M
Database Writer "updates" some existing records when option is disabled
Re: Database Writer "updates" some existing records when option is disabled
The problem was fixed and released last year.
If you are not using the database editor plugin to modify the aircraft records then you need to ensure that the UserString1 is not set to "Missing". The issue, or at least the one that I fixed, was that this field is used as a marker for placeholder aircraft records created from missing online lookups and the database editor plugin wasn't clearing the field when a manual edit was applied.
If you are not using the database editor plugin to modify the aircraft records then you need to ensure that the UserString1 is not set to "Missing". The issue, or at least the one that I fixed, was that this field is used as a marker for placeholder aircraft records created from missing online lookups and the database editor plugin wasn't clearing the field when a manual edit was applied.
Re: Database Writer "updates" some existing records when option is disabled
Thanks for the information regarding the "UserString1" field. However, the database updating issue I have seen in the past does not seem to be remedied by updating the "UserString1" field. I wrote a lengthy observation below. However, I think it can be summarized very easily. As far as I can tell, VRS (2.4.2) seems to ALWAYS update ALL aircraft records for approximately the first second after VRS is started as long as the Database Writer Plugin is enabled regardless of the plugin settings.
LITTLE MORE DETAIL
It appears that for approximately the first second after VRS is started, VRS will ALWAYS update aircraft in the BaseStation.sqb file from the online source because it ignores some settings/rules intended to prevent the database update from occurring. For example, VRS ignores the following settings/rules during the first second:
- "Overwrite details on existing aircraft" is unchecked in the Database Writer Plugin.
- The LastModified date of the aircraft is less than 28 days old.
It is important to know that each time VRS is started it will affect whatever aircraft it starts tracking in the first second. If VRS is restarted a number of times, VRS will continually modify more and more aircraft that should not have been modified.
TO REPLICATE THE BUG:
- Have VRS running at a time with many aircraft in the area.
- Make sure the Database Writer Plugin is enabled and verify these options:
- Change these three fields for ALL aircraft in the database like so:
- Start VRS using the newly modified BaseStation.sqb file.
- Bring up a web browser (preferably in incognito mode to prevent cached data from appearing) to view the VRS webpage.
- In the VRS webpage, pause VRS (click "Pause" at the top of the list of aircraft).
- In the VRS webpage, click Menu -> Options -> List, and make sure the list is only sorted by "Time Tracked".
At this point, a good portion (maybe half) of the aircraft might have had their Operator Codes of 'ZZZ' modified/updated with the Operator Codes from the online source when they should not have been. Aircraft that were not tracked until about one or more seconds after VRS was started probably did not have their Operator Codes updated. (I have a screenshot of my own experiment results. But, there is not a convenient way of posting it here. In my experiment, about 40 aircraft were modified.)
FURTHER INVESTIGATION WITH THE LAST MODIFIED DATE
The date I used above happens to be the current date today as I am writing this (2019-04-04). This was done intentionally as it further shows that not only should have VRS not updated the aircraft because the "Overwrite details on existing aircraft" option was unchecked, but VRS should have also not updated the aircraft because none of the aircraft had a LastModified date older than 28 days. Regardless if "Overwrite details on existing aircraft" is checked or not, VRS should not have updated existing aircraft less than 28 days old. The same experiment has been done with only all of the LastModified dates changed to the current date and the "Overwrite details on existing aircraft" option checked. The bug still updated aircraft in this situation as well.
WORKAROUND FOR THE BUG:
Here is one workaround to keep VRS from updating the database that always works for me:
- Whenever shutting down VRS and just prior to shutting down VRS, make sure to disable the Database Writer Plugin (uncheck "Enable" in the Database Writer Plugin options).
- The next time VRS starts up, wait a few seconds after VRS has completely loaded up.
- Re-enable the Database Writer Plugin (check "Enable" in the Database Writer Plugin options).
LITTLE MORE DETAIL
It appears that for approximately the first second after VRS is started, VRS will ALWAYS update aircraft in the BaseStation.sqb file from the online source because it ignores some settings/rules intended to prevent the database update from occurring. For example, VRS ignores the following settings/rules during the first second:
- "Overwrite details on existing aircraft" is unchecked in the Database Writer Plugin.
- The LastModified date of the aircraft is less than 28 days old.
It is important to know that each time VRS is started it will affect whatever aircraft it starts tracking in the first second. If VRS is restarted a number of times, VRS will continually modify more and more aircraft that should not have been modified.
TO REPLICATE THE BUG:
- Have VRS running at a time with many aircraft in the area.
- Make sure the Database Writer Plugin is enabled and verify these options:
- "Save online lookups in database" is checked.
- "Overwrite details on existing aircraft" is not checked.
- Change these three fields for ALL aircraft in the database like so:
- OperatorFlagCode = 'ZZZ'
- LastModified = '2019-04-04 00:00:00'
- UserString1 = NULL
- Start VRS using the newly modified BaseStation.sqb file.
- Bring up a web browser (preferably in incognito mode to prevent cached data from appearing) to view the VRS webpage.
- In the VRS webpage, pause VRS (click "Pause" at the top of the list of aircraft).
- In the VRS webpage, click Menu -> Options -> List, and make sure the list is only sorted by "Time Tracked".
At this point, a good portion (maybe half) of the aircraft might have had their Operator Codes of 'ZZZ' modified/updated with the Operator Codes from the online source when they should not have been. Aircraft that were not tracked until about one or more seconds after VRS was started probably did not have their Operator Codes updated. (I have a screenshot of my own experiment results. But, there is not a convenient way of posting it here. In my experiment, about 40 aircraft were modified.)
FURTHER INVESTIGATION WITH THE LAST MODIFIED DATE
The date I used above happens to be the current date today as I am writing this (2019-04-04). This was done intentionally as it further shows that not only should have VRS not updated the aircraft because the "Overwrite details on existing aircraft" option was unchecked, but VRS should have also not updated the aircraft because none of the aircraft had a LastModified date older than 28 days. Regardless if "Overwrite details on existing aircraft" is checked or not, VRS should not have updated existing aircraft less than 28 days old. The same experiment has been done with only all of the LastModified dates changed to the current date and the "Overwrite details on existing aircraft" option checked. The bug still updated aircraft in this situation as well.
WORKAROUND FOR THE BUG:
Here is one workaround to keep VRS from updating the database that always works for me:
- Whenever shutting down VRS and just prior to shutting down VRS, make sure to disable the Database Writer Plugin (uncheck "Enable" in the Database Writer Plugin options).
- The next time VRS starts up, wait a few seconds after VRS has completely loaded up.
- Re-enable the Database Writer Plugin (check "Enable" in the Database Writer Plugin options).
Re: Database Writer "updates" some existing records when option is disabled
I've had a look at the code and tried to reproduce it following your steps but so far I can't see anything in the code (or at least the bits of code involved with this) that would try to use an object before it's been initialised from settings, nor anything that would accidentally update an existing aircraft instead of creating a new one. I've also been unable to reproduce the issue using your instructions. However, there's only a few aircraft around at the moment so I'll try again at the weekend when things are busier.
In your instructions to reproduce the bug you have the creation of new aircraft records switched on and updates switched off.
When you have a look at the database after having reset everything to operator ZZZ / last updated = today, what do you see for the created date on the records that don't have an operator of ZZZ after the restart? Does the created date match the updated date? If it's a brand-new aircraft record then the two dates will be the same.
In my test I see two non-ZZZ records after the restart and for each of them the created and updated dates are the same. The SQB I'm using is a copy that I took with the flights taken out - when I look in the original for those two aircraft neither of them are there so they are genuine new records.
Since I started this (I've been looking at bits of code while I've been framing this reply) I've left it running a further 25 minutes and I have another 6 non-ZZZ aircraft records. All of them have matching created and updated dates and don't exist in the original SQB. This is in a BaseStation.sqb that has ~169,000 aircraft records and at a fairly quiet time of night, so it's maybe not as unusual as you might imagine to pick up new aircraft.
That said there's a lot going on with online lookup caching. I'll try again at the weekend and see if I can get a bunch of records to update then.
In your instructions to reproduce the bug you have the creation of new aircraft records switched on and updates switched off.
When you have a look at the database after having reset everything to operator ZZZ / last updated = today, what do you see for the created date on the records that don't have an operator of ZZZ after the restart? Does the created date match the updated date? If it's a brand-new aircraft record then the two dates will be the same.
In my test I see two non-ZZZ records after the restart and for each of them the created and updated dates are the same. The SQB I'm using is a copy that I took with the flights taken out - when I look in the original for those two aircraft neither of them are there so they are genuine new records.
Since I started this (I've been looking at bits of code while I've been framing this reply) I've left it running a further 25 minutes and I have another 6 non-ZZZ aircraft records. All of them have matching created and updated dates and don't exist in the original SQB. This is in a BaseStation.sqb that has ~169,000 aircraft records and at a fairly quiet time of night, so it's maybe not as unusual as you might imagine to pick up new aircraft.
That said there's a lot going on with online lookup caching. I'll try again at the weekend and see if I can get a bunch of records to update then.
Re: Database Writer "updates" some existing records when option is disabled
Also, if you do have genuine updates rather than new records, when you search for the aircraft by registration do you get multiple copies back where the ModeS field differs between them by case? E.G. one is "ABC123" and the other is "abc123"?
BaseStation.sqb's schema has case sensitive collation switched on for the ModeS field but VRS makes the assumption in a lot of places that ICAOs are always uppercase. If your feed has them in lowercase then the lookup will probably be saving them in uppercase while your feed is *maybe* saving in lowercase... I thought everything was getting normalised to uppercase nowadays but I don't have any lowercase feeds so I'm relying on unit tests to confirm that, there could well be a situation where lowercase ICAOs are saved.
If that is what's happening then you'll probably find your edits are untouched on the lowercase version of the ICAO and the uppercase version just has the lookup details on it.
BaseStation.sqb's schema has case sensitive collation switched on for the ModeS field but VRS makes the assumption in a lot of places that ICAOs are always uppercase. If your feed has them in lowercase then the lookup will probably be saving them in uppercase while your feed is *maybe* saving in lowercase... I thought everything was getting normalised to uppercase nowadays but I don't have any lowercase feeds so I'm relying on unit tests to confirm that, there could well be a situation where lowercase ICAOs are saved.
If that is what's happening then you'll probably find your edits are untouched on the lowercase version of the ICAO and the uppercase version just has the lookup details on it.
Re: Database Writer "updates" some existing records when option is disabled
I ran another experiment tonight to focus on what you wanted me to look for. At the time of the experiment, I also did not have many planes (~40-45). Here are a few things to report:
1. The experiment behaved the same with the same updating issue.
2. The first 24 aircraft that VRS starting tracking at the immediate start of VRS got improperly updated. (All subsequent aircraft kept their 'ZZZ' codes.)
3. I confirmed that none of the 24 were new aircraft at the time of the experiment. (Newest plane was 2019-04-01.)
4. I confirmed every ModeS of every aircraft in the SQB was and is capitalized.
5. I found no duplicate ModeS entry ("ABC123" vs. "abc123")
6. The FirstCreated dates for all 24 updated aircraft ranged from '2018-07-24' to '2019-04-01'. All LastModified dates were nearly identical with each other. In fact, all of the LastModified dates/time were either '2019-04-04 22:32:25' or '2019-04-04 22:32:30'. I think this was about the time I shut down VRS.
I hope this is some of the information you were looking for. Let me know what else you need from me. I am fine with doing the workaround that I already described. But, I know you probably would like to get to the bottom of this.
Oh, FWIW, I did a quick experiment with a newly created database just to make sure somehow the database I normally use is not at fault. The newly created database also behaved the same with the same updating issue.
1. The experiment behaved the same with the same updating issue.
2. The first 24 aircraft that VRS starting tracking at the immediate start of VRS got improperly updated. (All subsequent aircraft kept their 'ZZZ' codes.)
3. I confirmed that none of the 24 were new aircraft at the time of the experiment. (Newest plane was 2019-04-01.)
4. I confirmed every ModeS of every aircraft in the SQB was and is capitalized.
5. I found no duplicate ModeS entry ("ABC123" vs. "abc123")
6. The FirstCreated dates for all 24 updated aircraft ranged from '2018-07-24' to '2019-04-01'. All LastModified dates were nearly identical with each other. In fact, all of the LastModified dates/time were either '2019-04-04 22:32:25' or '2019-04-04 22:32:30'. I think this was about the time I shut down VRS.
I hope this is some of the information you were looking for. Let me know what else you need from me. I am fine with doing the workaround that I already described. But, I know you probably would like to get to the bottom of this.
Oh, FWIW, I did a quick experiment with a newly created database just to make sure somehow the database I normally use is not at fault. The newly created database also behaved the same with the same updating issue.
Re: Database Writer "updates" some existing records when option is disabled
Just to double-check - you're running version 2.4.2 of VRS?
Re: Database Writer "updates" some existing records when option is disabled
Correct. Version 2.4.2
Running on Raspbian Stretch on Rpi 3B+.
You might want to bring up VRS with your browser (I use Chrome) in incognito mode. I actually have no idea how much it will affect the results of my particular experiment, but I have found it tricky to get the most refreshed data from VRS - even after I do a CTRL+F5 - unless I bring the browser up in incognito mode. For example, when I create updated Operator Flag bmp's, even a CTRL+F5 refresh will bring up the older cached bmp file instead of the new one I just updated.
I created an imgur account to post the pic of my results here. Hopefully the pic will post just so I can help show evidence that I am not crazy!
Running on Raspbian Stretch on Rpi 3B+.
You might want to bring up VRS with your browser (I use Chrome) in incognito mode. I actually have no idea how much it will affect the results of my particular experiment, but I have found it tricky to get the most refreshed data from VRS - even after I do a CTRL+F5 - unless I bring the browser up in incognito mode. For example, when I create updated Operator Flag bmp's, even a CTRL+F5 refresh will bring up the older cached bmp file instead of the new one I just updated.
I created an imgur account to post the pic of my results here. Hopefully the pic will post just so I can help show evidence that I am not crazy!
Re: Database Writer "updates" some existing records when option is disabled
The updates happen regardless of whether anyone is looking at the web site.
The Pi doesn't have a real-time clock. Has the clock been set before you start VRS? Are you running VRS as part of a start-up script or will it exhibit the same behaviour well after the time has been established?
The Pi doesn't have a real-time clock. Has the clock been set before you start VRS? Are you running VRS as part of a start-up script or will it exhibit the same behaviour well after the time has been established?
Re: Database Writer "updates" some existing records when option is disabled
I understand the database is getting updated regardless. I only mentioned looking at the browser because it is quicker to check than the database.
Yes, the Raspberry Pi clock is always set long before VRS is started. It is not part of a startup script. (Side note: I actually looked into this before and have been unsuccessful in getting VRS to start at boot.) In fact, the RPi is rarely rebooted. For almost all of my tests, the length of time since the most recent boot of the RPi and before VRS is started/restarted can be measured in days.
I decided to try to remove the RPi variable. I dug up an old desktop computer running Lubuntu. I installed VRS using the same method and with the same settings as the VRS on my RPi. I ran one quick test with the same test conditions and it passed with flying colors. The only aircraft that got updated were new planes. Not one single aircraft got improperly updated.
Yes, the Raspberry Pi clock is always set long before VRS is started. It is not part of a startup script. (Side note: I actually looked into this before and have been unsuccessful in getting VRS to start at boot.) In fact, the RPi is rarely rebooted. For almost all of my tests, the length of time since the most recent boot of the RPi and before VRS is started/restarted can be measured in days.
I decided to try to remove the RPi variable. I dug up an old desktop computer running Lubuntu. I installed VRS using the same method and with the same settings as the VRS on my RPi. I ran one quick test with the same test conditions and it passed with flying colors. The only aircraft that got updated were new planes. Not one single aircraft got improperly updated.