richb, you really made me think... and search for "Fire" in all example pages. No need to say that I couldn't find "Fire" anywhere...

I'm reliefed now, seeing that it was a translation error :-D

Make links look more pretty using css:
https://forum.openwrt.org/viewtopic.php … 43#p272643
https://forum.openwrt.org/viewtopic.php … 62#p272662

Findings:

Database Size vs. real data

Database:
/var/www/demowiki $ du data/meta/data.sqlite3
4080    data/meta/data.sqlite3

Real data:
/var/www/demowiki $ du -s data/pages/toh/hwdata/
4324    data/pages/toh/hwdata/

-> real data (the dataentry pages) goes almost 1:1 into the database.

Performance (on rpi2, 900MHz, read/write micro SD card, only 1 core used by php)
- creation of datatable long toh (all columns we have): 17 sec
- creation of datatable shortest toh (minimum columns): 6 sec
- creation of datatable all urls: 11sec
- creation of datatable Firmware urls: 5sec
- creation of datalist long column/comma separated: 16 sec
- creation of dataclouds: <1 sec

My hopes are: The OpenWrt wiki is running on something more powerfull than a rpi2 (more MHz, usage of more cores for php / pthread support iirc).

In the demowiki:
- Started to fiddle around with css and succeeded to increase the width of the left column of the dataentries and to tame the url's width. -> "---- dataentry ----" has become "---- dataentry techdata ----"; with the additional "techdata", you can refer to it in css. Nice!
- added some pictures to the media:toh:d-link section and the dir-505 datapage.
Effect of the 's':
Picture_img100s -> 1pic ok, 2 pics ok -> all pics will be shown
Picture_img100 -> 1 pic ok, 2 pics nok -> exactly one picture will be shown when exactly one link is present; no pictures will be shown of 2 links or more are present.

It doesn't really matter as long as it's available in some form of "raw" format, separated by comma or something else. You could then use that to convert to some other suitable format, e.g. json or something else.

Hi ralph,

1) The "datalist" output of the dokuwiki data plugin delivers you html output, i.e. an unordered list (<ul>), where each list item equals one line / one device in the toh. The datafields in this list item can be separated by almost anything you want. However, strictly seen, this is not the export you imagine, right?

2) The data stored with the data plugin is saved to an sqlite db, which can be queried as you like.
I have no clue of sqlite, therefore support from someone else would be required to implement this export functionality directly via sqlite queries.

Example for 1):

<div class="dataaggregation"><ul class="dataplugin_list "><li><div class="li">Arcadyan / Astoria; ARV7506; ?; WIP; Lantiq XWAY Danube; lantiq; 2x333; 8; 64; 4x 100M + ADSL2+ + 2x FXS; ?; Yes; Ralink RT3060F; b/g/n; ; No; ?; ?; <a href="/demowiki/doku.php?id=toh:arcadyan:arv7506" class="wikilink2" title="toh:arcadyan:arv7506" rel="nofollow">arv7506</a>; <a href="/demowiki/doku.php?id=toh:hwdata:arcadyanastoria:arcadyanastoria_arv7506" class="wikilink1" title="toh:hwdata:arcadyanastoria:arcadyanastoria_arv7506">Techdata</a>; <a href='http://forum.openwrt.org/viewtopic.php?id=56521' class='urlextern' rel="nofollow">http://forum.openwrt.org/viewtopic.php?id=56521</a>; <a href='http://wikidevi.com/wiki/yourdevice' class='urlextern' rel="nofollow">http://wikidevi.com/wiki/yourdevice</a>; <a href='http://yourbrand.com/yourdevice' class='urlextern' rel="nofollow">http://yourbrand.com/yourdevice</a>; <a href='http://yourbrand.com/yourdevice/stockfirmware' class='urlextern' rel="nofollow">http://yourbrand.com/yourdevice/stockfirmware</a>; <a href='https://downloads.openwrt.org/chaos_calmer/15.05-rc1/ar71xx/generic/openwrt-15.05-rc1-ar71xx-generic-dir-505-a1-squashfs-factory.bin' class='urlextern' rel="nofollow">https://downloads.openwrt.org/chaos_calmer/15.05-rc1/ar71xx/generic/openwrt-15.05-rc1-ar71xx-generic-dir-505-a1-squashfs-factory.bin</a>; <a href='https://downloads.openwrt.org/chaos_calmer/15.05-rc1/ar71xx/generic/openwrt-15.05-rc1-ar71xx-generic-dir-505-a1-squashfs-sysupgrade.bin' class='urlextern' rel="nofollow">https://downloads.openwrt.org/chaos_calmer/15.05-rc1/ar71xx/generic/openwrt-15.05-rc1-ar71xx-generic-dir-505-a1-squashfs-sysupgrade.bin</a>;  Comments Geteilte Freude ist doppelte Freude, geteilter Schmerz ist halber Schmerz.</div></li></div>
</div>

I included the "Conventions for dataentry values" in the datapages.
Example see demowiki -> DIR-505 Techdata page -> Conventions added below the dataentry.
I like the include feature, it enables modularity

With the conventions included below the dataentry, we could drop the # comments in the dataentries.
Just a suggestion, in order to keep the dataentry pages small and increase modularity.

@zo0ok: for the next dataimport, can you please:
- change 'Device Techdata_pageid' to 'Device Techdata_hidden'
- include at the top: {{page>meta:infobox:dataentry_permanote&noheader&nofooter&noeditbtn}}
- include at the bottom: {{page>:template_dataentry#conventions_for_dataentry_values&nofooter&noeditbtn}}
(see also http://wiki.openwrt.org/toh/dataentry_template#usage where to place those)
- change to "---- dataentry techdata ----" (preparations for css usage)
- have a look at http://wiki.openwrt.org/toh/dataentry_template#usage, maybe I forgot something.

Take time, no need to hurry.

I just did a quick tryout: Editing the Techdata page of via the custom edit button (left edit button below the dataentry).
Conclusion: The comments are damn helpful. Quick instructions in the comments, detailed instructions in the conventions-include.

Other thought on this?

Done:
https://dl.dropboxusercontent.com/u/906 … index.html
https://dl.dropboxusercontent.com/u/906 … ikitxt.tgz

Ok?

I removed the fake values that were there just to slow down your RPi 2.
If you want them I generate a new tgz for that, but I dont want to publish such nonsense data to the above page

Oh - now either you were too fast with creating the data or I was too slow with writing this post
Anyway, thanks for the update!

Possibly unexpected request: Remove the "Status_" column.
Looking at the data in the demowiki I realized: You were right, we can absolutely live without it.
- Filtering for whatever we define as 'supported' can be done without the Status column
- Change of definition of 'supported' can easily be done via changing the filter criteria, rather than changing the datapages

And another request: Please copy values of "Supported Since Rel" to "Supported Current Rel". That's the best guess we can make today. Info should be updated later (15.05 final?) on the datapages, together with adding the links to the openwrt image downloads.

Two changes done, and data upated. Same old links as yesterday.

Import into demowiki done!

After a week of not doing much of anything (except some thinking) and seeing that there are already people searching the old toh with little success: What do you think of a testrun in the real openwrt wiki?

Our data is not bad, columns are defined, "customers" are waiting for results.

Two issues I see:
- Filtering for '?' doesn't seem to be possible (need for renaming '?' to 'unknown' or similar). I already asked in the dokuwiki forum, but no answer yet. I have a slight indication that this might be a bug, hence thinking of filing a bug report, but this may take some time to resolve.
- Filtering for '>32' doesn't work if manually entered in the filter headers (workaround: Filter via the tag-clounds; not exactly the same functionality, but works)

What's your opinion on this? Should we go for it or do we need more refinement first?
What do you think about the two issues above?

As a lurker on this thread, I would say that it would be helpful if you were to bring this up live as soon as possible. The TOH in the link given above is far more useful than the current one, even if you think further refinements are required.

https://dl.dropboxusercontent.com/u/906 … index.html

And thanks very much for the work you have done on this.

(Last edited by lizby on 28 Jun 2015, 15:07)

It seems we need a plan... I will try to invent something:

Prerequisites to deploying solution to wiki.openwrt.org

[ ] All plugins installed in wiki and all configuration done
[ ] tmo26 given enough access to bulk import 900+ txt files
[ ] A few tables with sane filters (Supported, Supported Long, All Devices)
[ ] How to edit entries 
[ ] How to create new entries
[ ] Block/lock editing of old ToH to prevent people from editing
[ ] Data values good enough
[ ] Sufficient functionality/performance
[ ] A "go" so tmo26 and I are not thrown out for destroying the wiki ;)
[ ] Things I have not though about???

Where do we stand?

I can imaging two scenarios:
1) We do all the above, and if it does not work out really well, we deal with it.
2) We do all the above, but only as a "test run" with the intention of really doing it in a few weeks. A release candiate. It requires that we

[ ] can delete/overwrite the 900+ data entries at a later point
[ ] we restrict editing of the the data entries, or
[ ] we clearly inform that the data entries will be deleted later, and editing is just for testing

Things that can go wrong?

[ ] Creating/editing entries the new way is not well received
[ ] Entries get edited and corrupted causing unexpected effects
[ ] Performance is unacceptable in production
[ ] Things I have not thought about...

Things that need to be done later:

[ ] Filling in links to openwrt-images
[ ] Filling in other information in dataentries that we currently dont have (mostly links)
[ ] Retiring the old ToH making it clearly obsolete and hard to find by mistake

A few more reflections from my perspective. The Data Entries support quite nice Data Tables and Data Lists. That will be a huge step forward compared to the current situations. However, the Data Tables may turn out to not be "nice enough" (tmo26 mentioned filtering limitations today, for example). Nevertheless, the Data Entries are great, regardless of limitations of Data Tables. It was quite easy for me to harvest information from the current ToH, it will be even easier to harvest information from a Data List in the future. Future opportunities can be

[ ] Improving the Data Tables of DocuWiki
[ ] Creating a Data-JSON plugin to DocuWiki
[ ] Build a JavaScript-based frontend (like the one I have provided), a Data List or Data-JSON as a backend.

I can change my mind if you have good arguments, but I suggest a full test deployment of everything to wiki.openwrt.org... encouraging people to use it and complain about it, but also informing that most likely it is just a test run and edited/added data will be lost.

It would be awesome to coordinate the new TOH launch with the CC final release!

I agree that we can move ahead (and I'll specifically respond to @zo0ok's proposal in a minute.)

- re: "?" - I think you are suggesting "unknown" as a replacement for "?", but I would prefer not to do this. I am reluctant to to use up the screen space for the gain in filtering. By this, I mean that a missing entry in the ToH indicates appears as a visual "hole". When you look quickly at the ToH, you can see where there's data, and where it's missing/not entered. If we used "unknown" everywhere, there would be a mass of black text without any differentiation.

- Alternative: what if we simply left unknown values blank? (I suspect it's easy to filter for "".)

- Another thought - it's easy enough for people to sort by clicking on a column, which moves similar items together.

re: >32 filter. I do not think this should hold us up.

Let's do it!

This is a *great* list. Thank you.

I think we couch this all as a release candidate. We could add the warning as part of the text included in each data entry, so we can remove the warning en masse when we're happy with it. I believe it's possible to tell which data entry pages have been recently modified, so we can fix them if we need to regenerate.

YES! What we have now is Good Enough (tm), and as @lizby says, it's *way* better than what's there. We can try it as-is, and carefully monitor what happens. (We're making the "trunk build" of the wiki. :-) If the performance of the primary ToH is dreadful or there's some other problem, we can back away, but it may be just fine.

We won't know "what can go wrong" until we try. (But we've created a nice list of things to watch out for.) This new work does not impact the core wiki: most readers will be blissfully unaware that we have done it.

@zo0ok's tasks are good, but I might change the sequence:

- First get permission to add the data entry's and other plugins. That may be a serious road block/time delay - we don't know.
- Getting permission also addresses the problem of getting thrown off the wiki. BTW I'll lock arms with you...
- Then add all the other stuff, as a release candidate.
- Once we understand how it's working, work to come up with clear instructions for new people to be able to follow the pattern.
- Pull the trigger and go live as the primary ToH.

Great Work, everyone!

A few more questions:

- Did we expect to include the "Conventions for dataentry values" on each new data entry page? I don't think it's necessary (and it certainly bulks up those pages). Perhaps we can use the bureaucracy plugin to automatically clone a template in our next pass through the ToH process.

- Is it possible to automatically update the links to firmware? If so, that would be quite useful, but since I don't think it's possible, we should not let that hold us back.

zo0ok: Great checklists, thanks for that!

I'll comment inside the codeboxes with preceeding '->'

Prerequisites to deploying solution to wiki.openwrt.org

[ ] All plugins installed in wiki and all configuration done -> Done
[ ] tmo26 given enough access to bulk import 900+ txt files -> should be done by the admins (I doubt the admins will give me full access :)
[ ] A few tables with sane filters (Supported, Supported Long, All Devices) -> Done, available in the demowiki
[ ] How to edit entries -> partially done, http://wiki.openwrt.org/toh/dataentry_template#permanent_note
[ ] How to create new entries -> Done, http://wiki.openwrt.org/toh/dataentry_template#usage
[ ] Block/lock editing of old ToH to prevent people from editing -> admins (isn't needed for tryout, is it?)
[ ] Data values good enough -> IMHO they are.
[ ] Sufficient functionality/performance -> Functionality OK (datatables/tagclouds in the demowiki); Performance in the real openwrt wiki: We will see.
[ ] A "go" so tmo26 and I are not thrown out for destroying the wiki ;) -> GOOD point! ;-)

Scenario #2:
2) We do all the above, but only as a "test run" with the intention of really doing it in a few weeks. A release candiate. It requires that we

[ ] can delete/overwrite the 900+ data entries at a later point -> easy, when datapages are placed in separate namespace (i.e. as planned today). Delete this namespace/folder, and you're done.
[ ] we restrict editing of the data entries, or
[ ] we clearly inform that the data entries will be deleted later, and editing is just for testing -> easy, by editing the permanent note (adding the beforementioned warning). Edit only the permanent note, and all pages using it will have that warning automatically included.

Things that can go wrong?

[ ] Creating/editing entries the new way is not well received -> low risk, editing and creating is easy (IMHO)
[ ] Entries get edited and corrupted causing unexpected effects -> low risk, if custom dataentry editor is used (you can not break the dataentry's structure with this editor); medium risk otherwise (editing via wiki pageedit) -> keep users from doing silly things -> permanent note on each datapage (http://wiki.openwrt.org/toh/dataentry_template#permanent_note)
[ ] Performance is unacceptable in production -> that's what I'm most interested: How this solution performs in the real wiki

We have workarounds for the '>32' and the '?' issue, with the latter being rather ugly IMHO.
Since it is so ugly, I'd consider this ?-workaround only as last resort or only valid for tryout.
I really want to be able to filter for unknown values, with '?' being the marker for unknown.

That's good news!

Future opportunities can be

[ ] Improving the Data Tables of DocuWiki -> content and appearance-wise (css magic)
[ ] Creating a Data-JSON plugin to DocuWiki -> I heard a bit of JSON, but don't know anything of it. [x] Tell me more!
[ ] Build a JavaScript-based frontend (like the one I have provided), a Data List or Data-JSON as a backend. -> sounds interesting!

Agree!

tmo26 needs priviliged access to wiki, or someone with access must step in and help.

Can we replace '?' with something else, like "unknown" or "???" that works better?

Same for me.

Filtering for '' can be done, but it seems not in the dynfilters. IMHO that's no big drawback, since the normal user shouldn't be too much interested in empty values anyway
The wiki-gardeners can easily create a datatable containing

filter : yourcolumn=''

and filtering for '' works.

That's another very good thought!

Since '?' currently produces errors, I'd tend to replace '?' by '' (nothing).
- Filtering for '' works
- Sorting of a column may suit the same purpose without filtering
-> Functionality "filtering for unknown values" is available.

However, we then have to think about a different way to mark questionable values ("? 500" (MHz)).

Sidenote regarding filtering: It makes creation of the datatables faster. The less content in the table, the faster you get it (ORLY? ;-). Tagclouds as shown in the demowiki are IMHO a nice and easy way to offer filter-choices and accordingly pre-filtered datatables to our users.

"Oh, yes, we also have the "long version" with all devices and all columns available, but I need to get it for you from the stock, so it will take a while. Stay seated, don't panic, I'll be back in 15sec with your datatable."

;-)

Anything with ? in it will produce errors, I'm afraid.
& could work...

It's a tryout. The complete, detailed help regarding allowed values can be included in the datapages with just one line of code. We can also leave it out, if regarded as too bulky.

Yesterday I started a small sidestudy with the Strata plugin, only for comparison. I think I can say: We're on the right way with the data plugin. Bureaucracy is next

We had that topic a while ago. Usage of "latest" like https://downloads.openwrt.org/latest/ was proposed, but since currently there's no sign of what OpenWrt version you are actually downloading, this is not a good idea.
However, this might change with 15.05, see https://downloads.openwrt.org/chaos_cal … x/generic/
If 15.05 images will keep the version in the filename, we can use "latest", which makes edits for future releases unneccessary. Nice!

Implemented in the demowiki -> see any 'Techdata' page

How about browsing through the source code of the Data Plugin (or whatever it is called) and search for '?' and replace it with '*' (or something else if '*' already means something)?