Fwurg

[Brend]

I have an idea to speed up tools' data loading times. This time it is not like the caching in that it "might speed things up in some cases" -- I can prove that it will speed up the tools in all cases. Of course, I will have to rework some of the tools' code, but when that's done the reworked tools will be blazingly fast. The thing is: there is a trade-off to be made. Since you people together are using the tools more than I do, I need your input on deciding...

The trade-off is this: I can make the tools much faster, but only if they use old data.

For example, I can make the planet viewer load in milliseconds if it uses data from roughly an hour ago. It might be that it loads data from a few minutes ago, but you might get data that is 59 minutes old. The speed-up isn't a sliding scale; so it doesn't matter if we choose to use data from about an hour old, or data from last week.

What I need your input on is:

• Which tools can use old data (and load very fast)?
• Which tools must absolutely use the newest data (but load slow)?

[Elmer]

It really depends on the tool. The population growth tool can be 9 weeks old, while the tax tool needs to be up to date every time.

I think the tax tool need to newest data. For me, I like to try some things with my economy and like to compare the results. For this I need the tax tool to be up to date. More general economy overview tools, like the Labour Distribution Chart, the Egocentric import/Export Visualisation and the Economy Analysis tools can actually be a week old. For these tools it doesn't matter if the data is from last week. The total products produced by the Union won't change that much over one week after all. Like: *Oh no, I added my zone, the Union should produce 100 products more than it is, we are producing 10 100 instead of 10 000!!* . It's irrelevant.

So we need the relevant update rate of the tools. Tools which you use for experimentation, like the tax tool, need to be up to date. The rest can be a week old, since their relevance are over a week time span. However, there is another catch: In the ideal situation, people update their system immediately after the turn reports are approved. Then the tools are updated and for the rest of the week, you have the current situation of your world. But players don't always immediately update their system. Self I sometimes update my system the days later, or even the day of the turn report. If a player update their system a day too late, they have the old data for another whole week long, which is a shame. Which means that the tools with a relevant update ratio of one week should update every day in order to be sure every player is up to date with their system.

Maybe it is nice to have a list of all the tools? I have the idea I am missing some tools, even when we add the power tool.

[Brend]

I think you basically have them covered, Elmer.

I agree that most tools can use data from yesterday... The only tool I normally use for experimentation is the Tax tool.

[Fedor]

Also for those few times where you do want recent data on those other tools, can you make some kind of update cache button?

[Brend]

Also for those few times where you do want recent data on those other tools, can you make some kind of update cache button?

I can see if I can work in something like that, but that comes with some problems. For one, the update might take anywhere from half a minute to several minutes -- which isn't exactly fast.

I'll see if I can make some kind of 'emergency refresh' option though... Maybe I can make it so that you can switch the tool to 'actual data' for your use only...

[Stuiter]

I agree that all except tax tool need to be fast. For the other tools and extra refresh button would be nice.
I think the power tool needs that button as zones can change every week.

[Brend]

I think the power tool needs that button as zones can change every week.

That could be worked around by using data from one or two days ago... It won't be up-to-date immediately, but it will get up to speed quickly.

Of course, I could also tweak things so the snapshot data will usually be captured a day or so after the turn reports have been approved. That way, the snapshot will contain fresh data for all people that update the wiki before the snapshot is taken.

[Brend]

Progress update: I have implemented a simple cache of historic data. I have dubbed the system D4-T4.

The location http://www.fwurg.net/cache/d4-t4/ contains the data, which you can have a look at if you want. It's in JSON format, so other scripts can easily read it; this means that you'll need some way to view it or your eyes will hurt (I use the JSONView firefox plugin). Note that these files contain a lot of data. Most tools will need to filter them somehow.

Updates occur around 04:30 each day, so the overview tools won't be that far behind the real deal. Currently, the list of included queries is rather small, I will expend it on demand.

Next up: Making a Javascript API that quickly and cleanly fetches the data and allows some preprocessing such as throwing away all zones not in the ic:smi-halek_system.

PS. Because these datafiles are just simple files your browser will also cache them, making things even faster :D

PPS. Note to geeks (including myself), I have added two Apache directives to the cache directory:

Code: Select all

Header set Access-Control-Allow-Origin "*"
Header set Cache-Control "max-age=28800, must-revalidate"


The first to allow cross-domain requests for these cached resources. The second to trigger the cache mechanism.