Extension talk:Cargo/Archive July to October 2021

Using DataTables Styling Options

Currently, the "dynamic table" display option uses mostly default options for DataTables display format. It is highly preferrable to be able to customize DataTables' default styling options. For example: set "compact" mode for DataTables.

Is it possible to enable these options in any way?

Centering Map at Particular Location in Query

Is there a way to force a centering of a map to a particular point in a cargo query? Usually the map is zoomed and positioned such that all the "hits" of the query show and are centered. However, one can force a zoom level to override the computed scale. Is there a way to force a center point as well? That can be done using {{#display_map|center= ...} but I haven't found a way to incorporate that into a cargo query. Parma100 (talk) 13:55, 8 July 2021 (UTC)

Unfortunately, no; but a "center" parameter (or "starting bounds", which is what Page Forms has) would be a good idea. Yaron Koren (talk) 15:28, 9 July 2021 (UTC)

Okay. Weekend project. I'll see what I can come up with. Parma100 (talk) 16:01, 9 July 2021 (UTC)

I've got centering working for all three mapping services. Adds a center= parameter and, if no zoom= is specified, adds a zoom parameter to flesh out the services' requirements. Now I'm working on Leaflet clustering. I figure I'll wait until both tasks are working before I submit them for merging. Parma100 (talk) 13:26, 13 July 2021 (UTC)

Leaflet clustering now works. I'll be initiating a merge request directly. Parma100 (talk) 03:45, 15 July 2021 (UTC)

That's great - I'm looking forward to both patches! Yaron Koren (talk) 00:35, 16 July 2021 (UTC)

Query time span for calendar doesn't quite fit

Hi there! First of all: I love this extension. It is quite powerful.

Though, I have to admit that I only worked with this extension for roughly four hours and already hit what I would call a bug.

The situation given is like this:

• I've got a simple table for events (Page, Start, End, Server (short name))
• I would like to display the table in a calendar format
• The calendar gets shown and some of the data displayed
• The events in the table may have a time span of a single day or a whole month
• Wiki is hosted by Miraheze so I don't have any file or database access

The problem is, some events are missing in the calendar. For example, I've got an event which takes place 2021-06-22 to 2021-07-20 and in the month July 2021 it doesn't get displayed. A similar problem occurs in weekly view, too.

The problem at hand (as far as I could analyze it) is with the way the data is being fetched. The GET looks like this: ?title=Special:CargoExport&tables[0]=EventPeriods&tables[1]=EventPeriods&join on[0]=&join on[1]=&fields[0]=Title,Start,End,Event&fields[1]=Title,Start,End,Event&where[0]=Server="SEA"&where[1]=Server="KR"&group by[0]=&group by[1]=&order by[0]=`Title`,`Start`,`End`,`Event`&order by[1]=`Title`,`Start`,`End`,`Event`&limit[0]=100&limit[1]=100&format=fullcalendar&color[0]=LightGreen&color[1]=LightCoral&text color[0]=black&text color[1]=black&start=2021-06-27&end=2021-08-08&_=1626992008486

As you can see, the initial query (and the previous&next buttons) set a limit (start=2021-06-27&end=2021-08-08) to the fetched data. In my eyes, my example event would partly take place in that limit. This might be a bug on fetching the dates within a given time frame. For me myself it would be enough if I could just set a higher threshold for start and end (is there a parameter for this?).

I hope I was able to describe my problem. Please don't hesitate to ask if you need further information or need to see our wiki. Thanks in advance!

You did in fact hit a bug! Sorry about that. I just checked in a fix for it - hopefully it will make its way to Miraheze before too long. Yaron Koren (talk) 17:39, 23 July 2021 (UTC)

Thanks for the quick fix! Miraheze merged the change into their productive system, too. I can confirm that the fix works fine. I'm looking forward to use Cargo more often in the future! Random visitor 12:48, 02 August 2021 (UTC)

Parsing order effect? Help welcome.

Using semantic mediawiki, I used to have a display programmed as :

I used [[{{#show: |?CoetusImage}}|x300px|link=]]
in : http://dufal.fr/w/index.php?title=Mod%C3%A8le:ItemSuite&oldid=17785

Now replaced by [[{{#cargo_query:tables=TOMEouLIVRE|fields=Image|where=ID='{{{RefCoetus|}}}'}}|x300px|link={{{RefCoetus|}}}]]
but I get a text version like

It seems to me that the parsing of [[Image: happens before the #cargo_query is extracted.

As a result, I cannot replicate with Cargo what used to work very well with SMW.

More specifically,

• when the TOMEouLIVRE defines the field Image as a string, I see the name of the picture but not the image itself.
• when the TOMEouLIVRE defines the field Image as a File, I see the image in the standard wikimedia format (framed, and with name under) which breaks my initial format.

Do I get something wrong ?

The first 4 highest level of page http://dufal.fr/wiki/Test use a series of formating of the field in TOMEouLIVRE table which I created to debug. The picture before the first title on that Test page is what I would expect to get (what I used to get with SMW's #show)

Many thanks.

Frederic.

Try adding "|no html" to the #cargo_query call. Yaron Koren (talk) 02:57, 8 August 2021 (UTC)

Also suffering from duplicate rows

I have recently forked a wiki from Fandom to my own host which makes use of extensive Cargo templates to store and query data. Unfortunately I'm finding that when I try save a page (via null edit) to create a new entry in a Cargo table the entry is being duplicated. An example can be seen on this page https://www.poewiki.net/wiki/Modifier:AbyssFireColdResistanceJewel1 When I run a null edit this entry gets duplicated and breaks subsequent queries.

Any idea why this might be happening?

Iamacyborg (talk) 21:15, 9 August 2021 (UTC)

I can't see this right now - right now there's no real data being stored for that page. Could you replicate the problem? Yaron Koren (talk) 03:14, 10 August 2021 (UTC)

You can see the dupes here https://www.poewiki.net/wiki/Special:CargoTables/mods Iamacyborg (talk) 07:18, 10 August 2021 (UTC)

Is there any data or additional information I could provide that might make debugging this easier? Iamacyborg (talk) 13:21, 10 August 2021 (UTC)

Okay, yes, I see the problem. That's very strange, and I don't know why it's happening. Does it only happen for null edits, or does it persist if you make some actual edit to a page? In any case, recreating the table will probably fix the issue, at least temporarily. Yaron Koren (talk) 14:01, 10 August 2021 (UTC)

I noticed in our templates we don't set any particular field as unique, might that be causing these dupes to happen? Iamacyborg (talk) 15:55, 10 August 2021 (UTC)

No, that shouldn't affect things - the duplicate check happens (or is supposed to happen) for all rows. Yaron Koren (talk) 21:40, 10 August 2021 (UTC)

Hmm, any idea how we might proceed? We have quite a few tables and having to rebuild them daily might get pretty slow, particularly if things break if someone inadvertently modifies or does a null edit of a page. Iamacyborg (talk) 07:22, 11 August 2021 (UTC)

I doubt you would have to rebuild them daily - you might only have to rebuild them once, unless null edits are planned to be a regular thing. Yaron Koren (talk) 19:42, 11 August 2021 (UTC)

Here's a weird one, we've found when editing pages that store data in our versions table that pages that end with a letter are not being duplicated https://www.poewiki.net/wiki/Special:CargoTables/versions You can see the duplicates in that table here https://www.poewiki.net/wiki/User:Iamacyborg/test Iamacyborg (talk) 07:17, 12 August 2021 (UTC)

After some further debugging, this seems related to Extension:PageImages, which when active, causes duplicate items to appear when a page is saved. Disabling the extension appears to have fixed the problem.

That's a great find - I just added a note about that to the "Known bugs" page. Yaron Koren (talk) 20:45, 19 August 2021 (UTC)

Is there any way to use both extensions? I started using PageImages fairly recently and have noticed more Cargo duplicates than usual. I don't know if it's related but it sounds likely. I like both extensions (though It's PageImages I'd give up if I had to choose one). Jonathan3 (talk) 22:35, 13 September 2021 (UTC)

@Yaron Koren: @Iamacyborg: We were having this issue over on Nookipedia and I found that performing a cargo_store without explicitly defining every field will lead to duplicates. I've created a task on Phabricator with more details. Thanks, ~SuperHamster Talk Contribs 14:38, 1 September 2021 (UTC)

Calender colours don't fit for a compound query

Hi, I hit a bug when using the calendar output combined with a compound query.

{{#cargo_compound_query:
|tables=EventPeriods;where=Server='SEA';fields=Title,Start,End,Event=_pageName;color=red;Comment=description;text color=white;name=Title;start=Start;end=End
|tables=EventPeriods;where=Server='KR';fields=Title,Start,End,Event=_pageName;color=blue;Comment=description;text color=white;name=Title;start=Start;end=End
|format=calendar}}

The compound query is defined to do two queries. Each query defines a colour for the calendar like color=red. The REST requests fits but the response doesn't fit as all results have the same colour.

Looking at the code I guess it's caused by the reuse of the variable $i in CargoExport.php around line 122 and 148.

This bug might be lower priority as it is possible to walk around this problem by defining the colour as field: CONCAT('#4b884b')=color

--Random Miraheze user 20:38, 18 August 2021 (UTC)

That's strange; you can see the "color" parameter working correctly here. Maybe this is some bug that has since been fixed? Yaron Koren (talk) 19:05, 19 August 2021 (UTC)

Now that's interesting. Maybe it's a problem on our host's side. Sadly, I can't debug the code he's using. Well, we'll work with the workaround then. For documentation reasons, here is the request:

https://counterside.miraheze.org/w/index.php?title=Special:CargoExport&tables[0]=EventPeriods&tables[1]=EventPeriods&join on[0]=&join on[1]=&fields[0]=Title,Start,End,Event=_pageName&fields[1]=Title,Start,End,Event=_pageName&where[0]=Server='SEA'&where[1]=Server='KR'&group by[0]=&group by[1]=&order by[0]=`Title`,`Start`,`End`,`Event`&order by[1]=`Title`,`Start`,`End`,`Event`&limit[0]=100&limit[1]=100&format=fullcalendar&color[0]=blue&color[1]=red&text color[0]=white&text color[1]=white&start=2021-05-30&end=2021-07-11&_=1629584478634

and the response:

[{"title":"Circuit Link System","start":"2021-06-09 21:00:00","end":"2021-06-30 21:00:00","color":"blue","textColor":"white","description":null,"url":"\/wiki\/Lee_Jisoo"},{"title":"Old Fear","start":"2021-07-07 21:00:00","end":"2021-07-28 21:00:00","color":"blue","textColor":"white","description":null,"url":"\/wiki\/Old_Fear"},{"title":"Sweet Promotion","start":"2021-06-23 21:00:00","end":"2021-07-07 21:00:00","color":"blue","textColor":"white","description":null,"url":"\/wiki\/Sweet_Promotion"},{"title":"Yoo Mina Recruitment Rate Up","start":"2021-06-30 21:00:00","end":"2021-07-21 21:00:00","color":"blue","textColor":"white","description":null,"url":"\/wiki\/Yoo_Mina"},{"title":"[[Fenrir Yoo Mina Growth Limited Mission]]","start":"2021-06-30 21:00:00","end":"2021-07-21 21:00:00","color":"blue","textColor":"white","description":null,"url":"\/wiki\/Yoo_Mina"},{"title":"Smell of the Game","start":"2021-06-21 19:00:00","end":"2021-07-19 19:00:00","color":"blue","textColor":"white","description":null,"url":"\/wiki\/Smell_of_the_Game"}]

In this example, the "Smell of the Game" is the one which should be red, but it is not. In any case, thanks for your time! --Random Miraheze user22:38, 21 August 2021 (UTC)

Using dynamic data for the allowed values field parameters in #cargo_declare?

Hi, is it possible to use dynamic data, something like a Category: for example, to feed the allowed values= field parameter?

My goal here is that the end user could add new values to the allowed values= using a Form (created using PageForms) instead of manually editing the template where the #cargo_declare is called.

Igorabsorto (talk) 07:36, 26 August 2021 (UTC)

Hi! That's not recommended, because you would then have to recreate the table every time the set of values changed. It's better to not have "allowed values" in the Cargo declaration, and instead handle it in the form, with "value from category" or something like that. Yaron Koren (talk) 15:35, 27 August 2021 (UTC)

Great, that makes sense. Thank you, Yaron! Igorabsorto (talk) 09:17, 30 August 2021 (UTC)

Date display (again)

I have a similar query to that asked by Parma100 in Date Display - I have a date field, that I would like to change the format of the output. If the stored date has only year or month/year, the format is great (e.g. 1919 / March 1919) respectively), but if it also has day then the format is `Y/m/d` (e.g. 1919/03/14) which is not particularly 'friendly' and could be ambiguous - I would like the display to be the same as MW default i.e. 14 March 1919. I have not been able to decode what the solution was in that topic.

Playing with the code sample in topic linked above I can force the date to `j F Y` but that stops dates with less precision working as required - I've not been able to get a value from `Fieldname__precision` to use in a condition.

This is what I currently have in my template:

{{#if:{{{MaidenFlight|}}}|
{{!}}-
! Maiden Flight
{{!}} {{#time:j M Y|{{{MaidenFlight|}}}}} }}

Many thanks, --Jaydublu (talk) 17:32, 28 August 2021 (UTC)

You could try something like this in a template:

{{#ifeq:{{#len:{{{value|}}}}}|4|
  {{#time:Y|{{{value|}}}}}|
  {{#if:{{#pos:{{{value|}}}|/}}|
    {{#time:j F Y|{{{value|}}}}}|
    {{#time:F Y|{{{value|}}}}}
  }}
}}

Jonathan3 (talk) 22:43, 13 September 2021 (UTC)

One template that attach to multiple tables

Hi, is there a way that the recreate table function of "all" attached templates recognize them ? Actually only the last call to attach is recognized.

I know in "Storing data" it is written not to attach to more than one table. But nevertheless it seems that atleast the template itself gets everything right (i.e. it shows at the top that it writes to table A, writes to table B and so on).

But at the special page cargo-tables only the last is shown to write to the corresponding attached table. And only this is recreated if we redo the tables via web or php command.

In our case we want to have some templates that combine several cargo tables/templates (i.e. attach to) and write some fixed values at some fields to the tables and only some as template parameters are passed. Or is there a better way to do this? Hope the question was somehow clear ;)

MediaWiki 1.35.3 (dcbca3e); PHP 7.3.29-1~deb10u1; PostgreSQL 12.4; Cargo 2.7.1 (29a184f)

Cheers, Brabi81 (talk) 13:24, 20 September 2021 (UTC)

In theory, there's nothing wrong with what you're doing; but in practice, I think there's a bug that prevents what you are trying to do from working. If that's true, then, until the bug is fixed, one thing you can do is have that template itself call other templates, each of which either declares or attaches to a single Cargo table. Yaron Koren (talk) 18:23, 20 September 2021 (UTC)

The thing with one template that call other templates works fine. Thanks for that idea! But I'll also try to keep one eye open to see if the bug is fixed in future patch notes. Thanks Brabi81 (talk) 05:43, 24 September 2021 (UTC)

Can't populate Cargo db fields with empty data

Hello, I've been running into this for weeks now. Basically, all Cargo fields appears to be mandatory. Running cargoRecreateData.php on a Table with a Page with some empty fields throws this:

Recreating data for Cargo table Tools in 5 seconds... hit [Ctrl]-C to escape.
Deleting and recreating table...
Handling template that adds to this table: Tool
Saving data for pages 1 to 3 that call this template...
Wikimedia\Rdbms\DBQueryError from line 1719 of /var/www/mediawiki-1.36.1/includes/libs/rdbms/database/Database.php: Error 1366: Incorrect integer value: '' for column `innovation_wiki_cargo`.`cargo__Tools`.`Display_width` at row 1 (localhost)
Function: Wikimedia\Rdbms\Database::insert
Query: INSERT INTO `cargo__Tools` (`Code`,`Type`,`Datasheet_file`,`Drawing_file`,`Other_file`,`Website`,`Buy_URL`,`CPU`,`GPU`,`Display_size`,`Display_resolution_width`,`Display_resolution_height`,`Display_width`,`Display_height`,`_pageName`,`_pageTitle`,`_pageNamespace`,`_pageID`,`_ID`,`Suppliers__full`) VALUES ('739H6-CH','Computer','','','','www.logitech.com','www.amazon.com','AMD Ryzen 9 590','Nvidia GeForce RTX 309','10','1280','480','','','My tool 02','My tool 02',0,70,2,'My contact 01')

#0 /var/www/mediawiki-1.36.1/includes/libs/rdbms/database/Database.php(1703): Wikimedia\Rdbms\Database->getQueryException()
#1 /var/www/mediawiki-1.36.1/includes/libs/rdbms/database/Database.php(1678): Wikimedia\Rdbms\Database->getQueryExceptionAndLog()
#2 /var/www/mediawiki-1.36.1/includes/libs/rdbms/database/Database.php(1244): Wikimedia\Rdbms\Database->reportQueryError()
#3 /var/www/mediawiki-1.36.1/includes/libs/rdbms/database/Database.php(2365): Wikimedia\Rdbms\Database->query()
#4 /var/www/mediawiki-1.36.1/includes/libs/rdbms/database/Database.php(2345): Wikimedia\Rdbms\Database->doInsert()
#5 /var/www/mediawiki-1.36.1/extensions/Cargo/includes/CargoUtils.php(1221): Wikimedia\Rdbms\Database->insert()
#6 /var/www/mediawiki-1.36.1/extensions/Cargo/includes/parserfunctions/CargoStore.php(431): CargoUtils::escapedInsert()
#7 /var/www/mediawiki-1.36.1/extensions/Cargo/includes/parserfunctions/CargoStore.php(159): CargoStore::storeAllData()
#8 /var/www/mediawiki-1.36.1/includes/parser/Parser.php(3356): CargoStore::run()
#9 /var/www/mediawiki-1.36.1/includes/parser/Parser.php(3041): Parser->callParserFunction()
#10 /var/www/mediawiki-1.36.1/includes/parser/PPFrame_Hash.php(263): Parser->braceSubstitution()
#11 /var/www/mediawiki-1.36.1/includes/parser/Parser.php(3230): PPFrame_Hash->expand()
#12 /var/www/mediawiki-1.36.1/includes/parser/PPFrame_Hash.php(263): Parser->braceSubstitution()
#13 /var/www/mediawiki-1.36.1/includes/parser/Parser.php(2879): PPFrame_Hash->expand()
#14 /var/www/mediawiki-1.36.1/includes/parser/Parser.php(1549): Parser->replaceVariables()
#15 /var/www/mediawiki-1.36.1/includes/parser/Parser.php(639): Parser->internalParse()
#16 /var/www/mediawiki-1.36.1/extensions/Cargo/includes/CargoUtils.php(593): Parser->parse()
#17 /var/www/mediawiki-1.36.1/extensions/Cargo/maintenance/cargoRecreateData.php(164): CargoUtils::parsePageForStorage()
#18 /var/www/mediawiki-1.36.1/extensions/Cargo/maintenance/cargoRecreateData.php(69): CargoRecreateData->recreateAllDataForTable()
#19 /var/www/mediawiki-1.36.1/maintenance/doMaintenance.php(112): CargoRecreateData->execute()
#20 /var/www/mediawiki-1.36.1/extensions/Cargo/maintenance/cargoRecreateData.php(173): require_once('/var/www/mediaw...')
#21 {main}
PHP Notice:  Uncommitted DB writes (transaction from Wikimedia\Rdbms\Database::begin) in /var/www/mediawiki-1.36.1/includes/libs/rdbms/database/Database.php on line 5605

What version of Cargo are you running? I think this problem has been fixed already. Yaron Koren (talk) 16:28, 22 September 2021 (UTC)

Hi, it's 2.8 --Carloposo (talk) 17:00, 22 September 2021 (UTC)

Hi Yaron, changing field types (Integer and Floats to String) fixes the issue. Does this ring a bell? BTW, the same issue is present with empty Date fields. --Carloposo (talk) 11:00, 23 September 2021 (UTC)

There are a few pages about this error and strict SQL mode, eg https://stackoverflow.com/questions/28606483/how-to-allow-empty-string-for-integer-in-mysql#39409037

I often save Cargo pages with empty date fields and it's ok, so maybe when Cargo creates tables it uses the database default which is different in your case. Jonathan3 (talk) 11:37, 23 September 2021 (UTC)

Thank you Jonathan, that made my day! --Carloposo (talk) 13:18, 23 September 2021 (UTC)

When I tried this out, all empty values were saved as "null", including for Integer fields. By coincidence, I just released version 3.0 of Cargo - I would try upgrading to this, because it could be that there was some recent change that is resulting in the different behavior. Of course, if you already modified your database settings, you might not able to know (and might not care) whether the code is acting differently. Yaron Koren (talk) 19:25, 23 September 2021 (UTC)

Thank you, Yaron. Is upgrading as easy as replacing the Cargo folder with the new one? --Carloposo (talk) 07:44, 24 September 2021 (UTC)

Yes. Yaron Koren (talk) 14:19, 24 September 2021 (UTC)

Quick way to switch in multiple replacement tables

Because of the duplicate row thing, I've set up a handy Windows shortcut using plink to log in and create replacement tables - but then it's a pain having to go to the website to click the ticks for several tables. It would be nice to be able to do them all at once.

• Maybe there could be a button on Special:CargoTables to do that?
• Or maybe a new command line script?
• Or maybe the existing script could have an extra parameter to switch in the replacement tables at the end?

Thanks Jonathan3 (talk) 10:01, 24 September 2021 (UTC)

I've just saved all the pages as bookmarks that can be opened all at once in different tabs, then I can click "Switch" for each. It's quick enough, and uses replacement tables. (I found that when I recreated the tables nightly without a replacement table, they still would have duplicates, which maybe was because of the clashing with the job queue problem.) Jonathan3 (talk) 20:55, 5 October 2021 (UTC)

Mediawiki still showing data even though Cargo database deleted

I am making a catalog for my books and have hit the data duplication bug. I have mediawiki 1.36.1 and Cargo 3.0, on PstgreSQL 13. In the process if trying to debug the issue I erased all the cargo data related to the catalog. However the data still shows up in mediawiki even after purging. I am not seeing where it is getting the data from. Can you please help? Also, is there any progress on the data duplication bug?

What do you mean by the data showing up? And what data duplication are you seeing? I personally haven't seen this problem in a long time. Yaron Koren (talk) 01:40, 7 October 2021 (UTC)

When I go to Catalog:Books, the book shows up and I can still view its details.

As for data duplication, when I edit a book, a duplicate appears in the sql database and in the cargo table.

I am working on a smallest case example which may help clarify.

What's on the page Catalog:Books? And an example for the duplication problem would help a lot, yes. Yaron Koren (talk) 13:08, 7 October 2021 (UTC)

Example of duplicate data

As promised, here is an example of my book catalog.

The main Book template is the following:

<noinclude>
This is the "Book" template.

{{#cargo_declare:_table=Books
|Title=String
|Subtitle=String
|Authors=List (,) of Page
|Genres=List (,) of Page
|Publishers= List (,) of Page
|Year_of_publication=Date
|Number_of_pages=Integer
|ISBN=Integer
|Owners=List (,) of Page
}}
</noinclude>
<includeonly>
{{#default_form:Book}}
{{#cargo_store:_table=Books
|Title={{{Title|}}}
|Authors={{{Authors|}}}
|Genres={{{Genres|}}}
|Publisher={{Publishers|}}
|Year_of_publication={{{Year of publication|}}}
|ISBN={{{ISBN|}}}
|Number_of_pages={{{Number of pages|}}}
|Owners={{{Owners|}}}
}}

{|
!Title
|{{{Title|}}}
|-
!Subtitle
|{{{Subtitle|}}}
|-
! Author(s)
| {{#arraymap:{{{Authors|}}}|,|x|{{#formredlink:form=Author|target=x}} }}
|-
! Genre(s)
| {{#arraymap:{{{Genres|}}}|,|x|{{#formredlink:form=Genre|target=x}} }}
|-
! Publisher
| {{#arraymap:{{{Publishers|}}}|,|x|{{#formredlink:form=Publisher|target=x}} }}
|-
! Year of publication
| {{{Year of publication|}}}
|-
! Number of pages
| {{{Number of pages|}}}
|-
! ISBN
| {{{ISBN|}}}
|-
!Owners
| {{#arraymap:{{{Owners|}}}|,|x|{{#formredlink:form=Owner|target=x}} }}
|}

{{DISPLAYTITLE:{{{Title}}} }}

[[Category:Books]]
{{DEFAULTSORT:{{#if:{{{Title|}}}|{{{Title}}}, {{PAGENAME}}}}}}
</includeonly>

The Book form is

<noinclude>
This is the "Book" form.
To create a page with this form, enter the page name below;
if a page with that name already exists, you will be sent to a form to edit that page.


{{#forminput:form=Book}}

</noinclude><includeonly>

{{{for template|Book}}}
'''Title:'''

{{{field|Title|input type=text}}}

'''Subtitle:'''

{{{field|Subtitle|input type=text}}}

'''Author(s):'''

{{{field|Authors}}}

'''Genre(s):'''

{{{field|Genres|input type=tokens}}}

'''Publisher:'''

{{{field|Publishers|input type=tokens}}}

'''Year of publication:'''

{{{field|Year of publication|input type=year}}}

'''ISBN:'''

{{{field|ISBN}}}

'''Number of pages:'''

{{{field|Number of pages}}}

'''Owner:'''

{{{field|Owners|input type=tokens}}}

{{{end template}}}

</includeonly>

I first create the tables which are created appropriately. I then create a very simple book with just a title, and everything else is blank.

In 'Category Books' only one book shows up as expected (sorry, cannot seem to be able to insert screenshot).

Category:Books
Jump to navigation
Jump to search
Pages in category "Books"

This category contains only the following page.
A

    A Book

But in the Books Table I have two books listed:

Table structure:

    Title - String
    Subtitle - String
    Authors - List of Page
    Genres - List of Page
    Publishers - List of Page
    Year_of_publication - Date
    Number_of_pages - Integer
    ISBN - Integer
    Owners - List of Page

This table has 2 rows altogether.

Recreate data.
Page 	Title 	Subtitle 	Authors 	Genres 	Publishers 	Year of publication 	Number of pages 	ISBN 	Owners
A Book 	A Book 								
A Book 	A Book 								

In the PostgreSQL database there are two entries with different _IDs:

_ID 	_pageName 	_pageTitle 	_pageNamespace 	_pageID 	Title 	Subtitle ......
1        a book	     a book	                 0      201     A Book	         ......
2        a book	     a book	                 0      201     A Book	         ......

This is the duplication problem I am trying to fix.

Your data structure looks fine. I didn't notice before that this was a PostgreSQL database - I'm guessing that is part of the issue; Cargo works with PostgreSQL, though it's much less well-tested than with MySQL, unfortunately. If you recreate the table, does the duplication go away? And if so, does it come back every time you resave the page? Yaron Koren (talk) 14:44, 11 October 2021 (UTC)

I went through the data structure line by line to simplify the problem to the bare minimum. The bug seems to be in the template. in #cargo_declare I declared a Subtitle, but in #cargo_store I did not use it. When I made that fix, everything works now. I tried it on a three line data structure and the same thing shows up. So I believe this is a bug and it should have raised an error or warning. --Lbundle (talk) 19:41, 11 October 2021 (UTC)

This issue with duplication if not every field was specified in #cargo_store turned out to be a bug that was added in February. I believe it's fixed now. Yaron Koren (talk) 21:00, 15 December 2021 (UTC)

Critical performance drop after version 2.4

Yaron, hello! After a long break, I decided to update my wiki. Its peculiarity is very large pages that generate a lot of records. After deploying the update on the auxiliary test server, I decided to try to overwrite one page. The result was either a 504 error, or only partial saving of records (from 150 to 300, although the page generates about 1500 records). I tried rolling back from the Master version to previous versions. It worked (and significantly faster) only when returning to version 2.4.

The test server runs Debian 11, MariaDB 10.5.11 and MW 1.35.4. --StasR (talk) 22:23, 11 October 2021 (UTC)

That's very interesting; it might be due to the data duplication check, which Cargo right now runs before every insert. With the old code, does it really save all 1,500 records? Yaron Koren (talk) 23:33, 11 October 2021 (UTC)

Here are the PageValues of the largest page. It contains 1991 entries. Yes, sometimes there are errors when recording. I have scripts that check that the values are written to the database to the end and without duplicates. But in general, everything works. --StasR (talk) 08:04, 12 October 2021 (UTC)

That's very interesting. If you're willing, it would be great if you could try it again with the latest code, but making a small change to the code, to see if the duplication check is really the issue. In the file includes/parserfunctions/CargoStore.php, just change this line (line 328 or so):

               $rowAlreadyExists = self::doesRowAlreadyExist( $cdb, $title, $tableName, $tableFieldValues, $tableSchema );

...to this:

               $rowAlreadyExists = false; // self::doesRowAlreadyExist( $cdb, $title, $tableName, $tableFieldValues, $tableSchema );

If you try that, please let me know if that fixes the performance problem. Yaron Koren (talk) 14:12, 12 October 2021 (UTC)

I checked the change on version 2.5. It didn't help. --StasR (talk) 14:51, 12 October 2021 (UTC)

As a result of these experiments, I lost the “Page Values” menu item. Do you have any ideas how to restore it? --StasR (talk) 15:21, 12 October 2021 (UTC)

It turns out that this happens after executing “git checkout 2.4”. --StasR (talk) 15:49, 12 October 2021 (UTC)

Okay, thank you for trying that. And thanks for clarifying that the problem came in version 2.5. Given that, I have a new theory as to what is causing the slowness: this change, which added the $cdb->lockForUpdate() call to CargoStore.php. Could you try commenting out that line in later versions, and see if that speeds things up? Yaron Koren (talk) 16:27, 12 October 2021 (UTC)

YESSSS! ) Checked for REL1_35 (aka 2.6) and for ‘master’ aka 3.0 (but I am not ready for ‘master’ due to changes in the storing of empty values). Thanks. --StasR (talk) 17:04, 12 October 2021 (UTC)

After a month of using the version 2.6, I can say with confidence that this version is faster and more reliable. I'm a little worried about what a commented-out line (lockForUpdate) might entail. --StasR (talk) 13:18, 17 November 2021 (UTC)

Okay, that's good to hear. I just removed that line from the code (which I had been planning to do anyway). I think it'll be fine - that line was only added about a year ago, and the code worked fine beforehand. Yaron Koren (talk) 15:16, 17 November 2021 (UTC)

Not show pages' namespaces on Special:Drilldown

Is this possible?

The problem (for me) currently is that they're all listed alphabetically according to namespace, which means that each column is like A, A cont'd, A cont'd...

Thanks. Jonathan3 (talk) 21:03, 12 October 2021 (UTC)

Bizarrely, today I have just remembered these things from 2017:

• Extension_talk:Cargo/Archive_January_to_February_2017#Not_display_namespace_in_List_of_Pages
• Extension_talk:Cargo/Archive_January_to_February_2017#Not_display_namespace_in_Special:Drilldown
• T157849: Add option to hide namespace in Special:Drilldown and/or query results
• T157754: Fix the way namespaces are treated in the headings of Special:Drilldown

These were two patches I submitted that you used, firstly to be able to hide the namespace and secondly to use alphabetic headings based on page name (ignoring namespace).

The first ($wgCargoHideNamespaceName) still works, but the second doesn't any more. Jonathan3 (talk) 09:12, 21 October 2021 (UTC)

Help with conceptual knowledge gap with regards to implementing Cargo

Very naive question—I have have read many of the "getting started" type guides (like the link below) and watched several videos to wrap my mind around how Cargo works and even produced a simple working example on my wiki but I am still confused by one point that I can't quite seem to find information on in these guides (maybe only puzzling to me). I don't understand how one is supposed to 'populate' the Cargo table once it has been declared and stored. For example, with a simple example like an info box with authors and books) or something like that, what if you had 100s of entries in a csv that you wanted to upload to define the relations in an Infobox—i.e. the actual "data"? Do these have to be manually entered one row at a time on the table viewer or through the form that is generated with a Cargo table? I tried playing around with importing csv's into the wiki as well the External Data extension (which is mentioned elsewhere in this discussion thread) but I'm not sure how to take advantage of either to populate data into an existing Cargo table. Any help or a point toward the right resources would be great.

https://workingwithmediawiki.com/book/chapter16.html

— Preceding unsigned comment added by Flyingratchet (talk • contribs) 15 October 2021

The Data Transfer extension may be what you need in this case. (Although maybe you've used that already, because you talked about importing CSVs.) Yaron Koren (talk) 00:10, 15 October 2021 (UTC)

@Flyingratchet: I don't understand how one is supposed to 'populate' the Cargo table once it has been declared and stored. The 'stored' part of this is the populating of the table, and is done by adding a Cargo-ified template to a page. When that page is saved, the data is added to the Cargo table. So, yes, the data does have to be 'manually' entered, and that's done on wiki pages and not via any Cargo-specific UI. Sam Wilson 10:35, 18 October 2021 (UTC)

Okay thank you everyone and thank you @Samwilson: ! That was a big part of my confusion. I naively and erroneously assumed that the Cargo table was the original source of truth and, in turn, it was populating all the respective templates on every wiki page. Now I see that you can modify a pages template and the table updates, or in turn you can edit the row in the table viewer and the template changes in the article. Slowly but surely I'm getting it. Flyingratchet (talk) 21:50, 20 October 2021 (UTC)

How to find the empty value "where" in lua

how to find the empty value "where", just like in the Drilldown, you can select the empty value "nothing" _none.

example

if frame.args ['class'] ~ = '' then
   args.where = args.where .. 'AND requires_class HOLDS LIKE "%' .. frame.args ['class'] .. '%"'
else
   need to find all empty "class" values
end

Thanks. ~~ Oleksii

Every list field gets a column in the main table named *__full, so I think it's possible to query the absence of any value with e.g.

args.where = args.where .. ' AND requires_class__full IS NULL'

Sam Wilson 05:34, 17 October 2021 (UTC)

How to not show a Template in Special:Drilldown

Hello, I understand that every Cargo table gets automatically added to the tabs in the Special:Drilldown page. Is there a way to not show some of them? Thank you. --Carloposo (talk) 15:47, 20 October 2021 (UTC)

I thought that there was a way to do that, but it looks like there isn't. Out of curiosity, is there any pattern to the tables you don't want shown, in terms of the set of fields, number of rows, etc.? Or is it just information that you don't want people to look at? Yaron Koren (talk) 16:05, 20 October 2021 (UTC)

You can get rid of all of them using CSS. Maybe you could similarly just make the nth one disappear? Or each one might even have its own id? Jonathan3 (talk) 22:15, 20 October 2021 (UTC)

Oh yeah, that's true. Or JavaScript, but CSS is easier. Yaron Koren (talk) 23:58, 20 October 2021 (UTC)

@Yaron: it's infos I want to store in Cargo tables but don't want people to look at. I'll probably query them via API, if it's possible, and make them available to other systems (e.g. to do some statistical analysis). @Jonathan: that's a good idea, thanks. --Carloposo (talk) 08:31, 21 October 2021 (UTC)

@Carloposo: sounds like the CSS or JS hiding method might work for you, but do remember that this doesn't actually make the data private at all, and that there are still a number of ways for users to see it. —Sam Wilson 08:35, 21 October 2021 (UTC)

@Sam: you are totally right. It would be very nice to be able to decide which table can be drilled and which not. Also @Yaron Koren for hiding bpmnData and ganttData from Drilldown, as I understand there are no customisation for them (e.g. hiding/renaming specific fields etc.)

This might not be ideal for you, but maybe you could mark every field of the relevant tables as "hidden" so that they don't become filters on Special:Drilldown and (I think) also don't show on Special:CargoTables. Jonathan3 (talk) 13:28, 22 October 2021 (UTC)

Can't save data into bpmnData and ganttData tables

Hello, I installed FlexDiagrams and run the maintenance scripts to create Gantt and BPMN tables. Then I created a page in each of the namespaces. No data is saved in the cargo tables (and no data pops up in the Drilldown tabs). Thanks. --Carloposo (talk) 10:12, 21 October 2021 (UTC)

What happens if you re-run those scripts - does the data show up then? Yaron Koren (talk) 14:08, 21 October 2021 (UTC)

It does, thanks @Yaron Koren --Carloposo (talk) 07:31, 22 October 2021 (UTC)

Okay, I just checked in a change to Cargo so that now, if a page is created or modified (or deleted), all the relevant "special tables", like _bpmnData and _ganttData, should get changed accordingly. Yaron Koren (talk) 17:17, 28 October 2021 (UTC)

SUM the output of COUNT(*) (SOLVED)

Hi Yaron,

I'm using the cargo_query to select a subset of a set of tables, with the goal to find out how many records satisfy that query. What I would like to do is using the SUM() function to estimate this. So something like explained at: https://stackoverflow.com/questions/14880080/mysql-sum-of-a-count-with-group-by-clause

This is how far I am:

{{
#cargo_query:tables=BibEntryCargo,_pageData
|join on=BibEntryCargo._pageName = _pageData._pageName
|fields=BibEntryCargo._pageName= _pageData._creationDate,COUNT(*)
|limit=5000
|where=  _pageData._creationDate LIKE '%2021%'  GROUP BY BibEntryCargo._pageName
|format=template|template=CountingCARGO
}}

However, I would like to sum the count(*), so something like SUM ( COUNT(*) ). Any idea to do such? Now the result of the above is a string of ones. See: https://csdms.colorado.edu/wiki/Test_cargo1

I tried something like:

|fields=BibEntryCargo._pageName= _pageData._creationDate,COUNT(*),SUM(COUNT(*))

but that gives the following error: "Error 1111: Invalid use of group function (localhost)".

I'm using MySQL 8.0.26, MW 1.36.2, Cargo (3.0; latest downloaded 2 days ago) and ONLY_FULL_GROUP_BY is enabled.

Hope you can give a hint, --Kettner2 (talk) 17:08, 24 October 2021 (UTC)

@Kettner2: It's weird that that GROUP BY syntax is even working — I thought it had to be a separate parser function parameter. Anyway, it sounds like you want a list of years and page/record counts; something like this might do it:

{{#cargo_query:tables = BibEntryCargo, _pageData
|join on  = BibEntryCargo._pageID = _pageData._pageID
|fields   = YEAR(_pageData._creationDate), COUNT(*)
|group by = YEAR(_pageData._creationDate)
|format   = template
|template = CountingCARGO
}}

Note that I've joined on page ID to avoid issues with page titles. —Sam Wilson 01:42, 25 October 2021 (UTC)

Thank you @Sam! This is exactly what I was trying to get, record counts per year; and your formulation works, see: https://csdms.colorado.edu/wiki/Test_cargo2 (added a line to indicate the year I'm after, e.g.:|where= _pageData._creationDate LIKE '%2021%' ).

I'm glad this worked. And yes, I too am surprised that "GROUP BY" worked within "where" - that should probably be considered a bug. Yaron Koren (talk) 17:50, 25 October 2021 (UTC)

Empty variables missing in cargo_query template results (Work around)

Hi Yaron, all,

One more question using #cargo_query. I'm using something like the following query to parse publication information to a template:

{{#cargo_query:tables=BibEntryCargo
|where = _pageName = 'something'
|default= 
|fields=_pageName, BibType,Title, Editors, Journal, Booktitle, Volume, Pages, URL, DOI, ISBN, Note, Firstname
|format=template
|template=sometemplate 
}}

Depending on the article type (book, journal paper, thesis, etc.) certain fields are empty. For example, a thesis might not have a DOI. In earlier versions of Cargo (or mysql), these empty fields would still be recognized and as such stayed as empty variables in my template that uses something like {{{1}}}, {{{2}}}, etc. However, when using the function above in mysql8+ and the latest version of Cargo, now empty fields are ignored, they are not passed on to the template. So for example, if the 3rd field is empty, so no "Title" is entered, then my variable {{{3}}} becomes "Editors". So my question is: is there a way that empty fields are still recognized, such that this information is passed to a template? Thanks! --Albert Ke (talk) 19:57, 25 October 2021 (UTC)

I can't replicate this problem, although I'm not using MySQL 8 - I'm using the equivalent of an earlier version. So maybe the MySQL version is the issue - although I can't see how that would affect things... Yaron Koren (talk) 23:37, 25 October 2021 (UTC)

Thank you Yaron. What is somewhat odd is that the default print statement does list the empty fields, but when the results are passed through a template the empty fields are ignored. See for example:

• Template: https://csdms.colorado.edu/wiki/Template:Sometemplate
• Page with the 2 queries, 1st default print statement, 2nd using a template: https://csdms.colorado.edu/wiki/Test_1
• Records that hold the data: https://csdms.colorado.edu/wiki/Reference:Reference-000002

In the above example, the 2nd query (using the template) has a value for the 4th parameter (Editors), while you see from the 1st query that doesn't use a defined print statement, the 4th parameter (Editors) is empty. Could it be that empty fields are not parsed correctly in the Carqo_query statement? --Kettner2 (talk) 02:03, 26 October 2021 (UTC)

OK, I found a work around that involves "named args=yes". Use this in the query. Then use named variables in your template and query. So the query would become something like:

{{#cargo_query:tables=BibEntryCargo
|where = _pageName = 'something'
|default= 
|fields=_pageName=page, BibType=bib,Title=title, Editors=eds, Journal=journal, Booktitle=booktitle, Volume=vol, Pages=pages, URL=url, DOI=doi, ISBN=isbn, Note=note, Firstname=firstname
|named args=yes
|format=template
|template=sometemplate 
}}

And then in your template use {{{page|}}}, {{{bib|}}}, etc. instead of {{{1|}}}, {{{2|}}}, etc. --Albert Ke (talk) 22:03, 26 October 2021 (UTC)

I wouldn't say this is "solved" yet, exactly - there's still a problem with the handling of unnamed/numbered arguments. If you're willing, it would be quite helpful if you could add the following line to the function displayRow() within the file /includes/formats/CargoTemplateFormat.php, around line 18:

print_r($row);

...and then see what is displayed on the page with the template query. I'm very curious to know if MediaWiki's DB-querying functions are not including fields that are blank in the results. Yaron Koren (talk) 00:10, 27 October 2021 (UTC)

Ok, the following is printed:

Array ( [_pageName] => Reference:Reference-000002 [BibType] => journalArticle [Title] => HYDROTREND: A CLIMATE-DRIVEN HYDROLOGIC-TRANSPORT MODEL FOR PREDICTING DISCHARGE AND SEDIMENT LOAD TO LAKES OR OCEANS [Journal] => Computers & Geosciences [Volume] => 24 [Pages] => 51–68 [URL] => https://linkinghub.elsevier.com/retrieve/pii/S0098300497000836 [DOI] => 10.1016/S0098-3004(97)00083-6 [Note] => Auto downloaded ref at: 2019-12-31 [Firstname] => Syvitski, James P.; Morehead, Mark D.; Nicholson, Murray; )

Seems like data is presented for the right fields for this print statement.... --Kettner2 (talk) 00:25, 27 October 2021 (UTC)

Okay, thank you. No, that's bad, actually - there should be an 'Editors' parameter there with a blank value, for example, but instead it's just missing from the array entirely (as I suspected was the case). I'll have to see if there's some way to get around this problem. Yaron Koren (talk) 00:53, 27 October 2021 (UTC)

Thanks for looking into this Yaron! I have changed the status from 'solved' to 'work around'. --Kettner2 (talk) 01:16, 27 October 2021 (UTC)

Section with a variable name?

Is it possible to have a section that the user names?

Yes, if you have the section (and its header) as fields of a template. Yaron Koren (talk) 23:37, 25 October 2021 (UTC)

Links continue to be displayed as plaintext

I just want to remind this bug, see [1] and [2]. It has no relation to Extension:ConfirmEdit (I have never used it on my current testing mediawiki installation).

I tried many ways how to change code of my template but without any success. BTW, this code is extremely simple:

* [{{{1}}} {{{2}}}]

Only workaround known to me is to purge cache of the page where the query is placed - which is very unsatisfactory. Any ideas how to solve this (no lua, please, as I use standard webhosting, not server hosting).

--Radouch (talk) 19:36, 30 October 2021 (UTC)

I get this too though purging the page hasn't bothered me. It's on a Cargo query using a template output format which displays images. The images appear as HTML instead, whenever the page is edited/saved, then as the actual images when the page is purged. I use ConfirmEdit but for account creation rather than page edits. I could try disabling it... Jonathan3 (talk) 20:25, 30 October 2021 (UTC)

As I mentioned, ConfirmEdit seems unrelated to this. For my case I just now "discovered" that I can use Concat, so now instead of template I use this query:

{{#cargo_query:
tables=Links
|fields=CONCAT('[',url,' ',_pageName,']')
|format=ul
}}

I hope thus I can replace most of possible "template-use" cases in the future. --Radouch (talk) 20:43, 30 October 2021 (UTC)

I created a simple query/template like yours above and get a similar result, even with ConfirmEdit disabled. I haven't quite worked it out (it's late here) but... The first time the page with the Cargo query is saved it shows HTML tags. This can be fixed by purging the page. The page will be "broken" again when the Cargo query is changed (e.g. changing the limit parameter). The page will not be "broken" again by a tiny edit to the page (e.g. adding a space character to the end of the page) but otherwise will break when edited (a few characters, or a space within the page). Sometimes just saving the page again fixes it but that seems unpredictable/rare. Jonathan3 (talk) 23:21, 30 October 2021 (UTC)

I think any query using the "template" output is affected in the same way, and the way to ensure it's parsed/displayed correctly is always to purge the page after editing/saving it. Jonathan3 (talk) 23:25, 6 November 2021 (UTC)