Extension talk:Cargo/Archive May to July 2023

Self-Inflicted Duplicate Issue

Putting the solution at the top!
Don't do this: My MediaWiki Page -> My Template #1 -> Cargo Query using Template #2 for display -> Template #2 stores Cargo data.
Re-using Template #2 caused rebuilding of the associated Cargo database to pick up on My Mediawiki Page as something to include in Cargo despite the exact same data being present in Template #2.
Thanks to Yaron Koren for patiently debugging my recursive template usage issue.

Original problem follows...
This isn't a recurrence of the duplicate rows issue that cropped up in 2021. I am 99.9% sure it's self-inflicted.

Steps

1. Import page data from Excel into MyCargoDB. All pages are prefixed with CARGO ("CARGO_MyPageName") because they're data, not to be looked at directly by readers.
2. The template MyTemplate queries data pulled into Cargo from the imported CARGO pages.
3. Page called MyPageName uses MyTemplate to display data pulled into Cargo from CARGO_MyPageName.
4. Rebuild MyCargoDB.
5. Now MyPageName using MyTemplate displays Cargo data twice.
6. The view in Cargo Tables of MyCargoDB shows CARGO_MyPageName AND now MyPageName which explains why it's appearing twice.
7. If this keeps up I'll have 150 duplicates.

I'm not sure what mistake I've made to create this circular import into Cargo.

SovereignGFC (talk) 15:06, 2 May 2023 (UTC)

That first step sounds odd - are you directly creating a table within the database? If so, you shouldn't do that - the data stored by Cargo should always reflect what's in the wikitext. Could that be the issue? Yaron Koren (talk) 02:56, 3 May 2023 (UTC)

Apologies, I am not directly creating a table. I import through an extension that creates wiki pages from said Excel sheet that match MyTemplate.

After importing, I immediately execute runJobs.php.

I then Recreate data on MyCargoDB.

If any MyPageName using MyTemplate that got put into MyCargoDB exists at that point, MyCargoDB gains an entry from MyPageName even though MyCargoDB was already populated by MyTemplate. SovereignGFC (talk) 22:02, 3 May 2023 (UTC)

Okay, never mind. Well, it's strange that pages that look simply like "MyPageName" are showing up as holding Cargo data - even though they're meant to only display, not store, Cargo data. What exactly is in the template "MyTemplate"? Yaron Koren (talk) 14:30, 4 May 2023 (UTC)

MyTemplate is NOT a Cargo-generated template. It is an actual page template that DOES contain Cargo queries using templates created by Cargo's Create a class. SovereignGFC (talk) 01:11, 5 May 2023 (UTC)

Well, does MyTemplate contain a call to #cargo_store? And if not, how are the pages that call it getting Cargo data stored for them? Yaron Koren (talk) 17:32, 5 May 2023 (UTC)

MyTemplate does not have calls to #cargo_store.

MyPageName -> MyTemplate -> Cargo query in MyTemplate -> Separate Cargo-specific pages/templates referenced in query. SovereignGFC (talk) 18:29, 5 May 2023 (UTC)

Alright. Well, barring more information about your setup, I'm totally stumped - I have no idea how pages like MyPageName could show up in Special:CargoTables. Yaron Koren (talk) 18:43, 5 May 2023 (UTC)

What additional information might be helpful? SovereignGFC (talk) 00:50, 6 May 2023 (UTC)

The contents of MyPageName and MyTemplate, I suppose. Yaron Koren (talk) 13:29, 7 May 2023 (UTC)

Header in dynamic table

How to change header in dynamic table?

Tamplate: {{#cargo_query: tables=Items |fields=CONCAT( '{{item icon|', Items.name, '}}' ) |format=dynamic table }}

Table header looks like this:

name, '}}' )
Item
Item

Thanks! -- Oleksii 212.80.47.83 06:58, 6 May 2023 (UTC)

Add something like "=ColumnName" after the final ")" - i.e., set an alias for the field. Yaron Koren (talk) 13:30, 7 May 2023 (UTC)

how to display format=dynamic table in Lua?

local args = {
where = '...',
limit='10',
format='dynamic table',
orderBy = '_pageName',
}

the word 'format' seems to be a Lua reserved word

--Oleksii 212.80.47.83 15:55, 7 May 2023 (UTC)

There might not be any way to display result formats from within a Lua Cargo query - I'm not sure. Yaron Koren (talk) 02:42, 8 May 2023 (UTC)

It looks like maybe Lua wouldn't mind you using "Format" (see https://stackoverflow.com/questions/5530039/make-a-reserved-word-an-identifier-variable-in-lua) - but Cargo seems to want "format". Jonathan3 (talk) 14:24, 29 June 2023 (UTC)

WITHIN statement error when a Hierarchy item has an apostrophe

I have a hierarchy definition in a template that includes the item "St. Mary's Anglican Church". When I try to query this with a WITHIN statement, I get an error that appears to split this at the apostrophe:

WHERE Collection IN ( SELECT `_value` FROM `cargo__CollectedIn__Collection__hierarchy` WHERE `_left` >= ( SELECT `_left` FROM `cargo__CollectedIn__Collection__hierarchy` WHERE `_value` = "St. Mary' ) AND `_right` <= ( SELECT `_right` FROM `cargo__CollectedIn__Collection__hierarchy` WHERE `_value` = "St. Mary' ) )s Anglican Church") ORDER BY `cargo__Collection`.`_pageID`,`cargo__Collection`.`_pageName` 

. When I put double-quotes around a string with an apostrophe in a WHERE clause this seems to work fine. Is this a bug or am I writing the query wrong? Thanks! Tenbergen (talk) 04:10, 9 May 2023 (UTC)

Sorry for the delay. That's definitely a parsing bug in Cargo. Yaron Koren (talk) 12:54, 7 June 2023 (UTC)

Do you think you will fix it? If not I will have to find a work-around to not use hierarchies. Would you be OK if I add a warning to the documentation that hierarchies will break for within statements if the contents contain an apostrophe? You have clean documentation for your extensions and I don't want to mess that up. 104.246.134.43 17:22, 9 July 2023 (UTC)

Recreate and 500 Line Bug Again

• MediaWiki 1.39.3
• Cargo 3.4.2

When I kick off re-create data from the Cargo tables page, it gets stuck at 500 lines.

When I import a spreadsheet to that same table/template with 500+ lines and execute runJobs.php, the table populates.

What additional diagnostic or debugging data can I provide? SovereignGFC (talk) 03:40, 13 May 2023 (UTC)

Sorry, I missed this before. Do you see any errors in the browser's JavaScript console? Yaron Koren (talk) 02:37, 29 May 2023 (UTC)

Now that the backup function (export to Excel which keeps my special character filled text intact) works thanks to the phpspreadsheet update, I was able to test this.

1. Open browser dev/debug tools.
2. Recreate data link on template page, Cargo view all tables page (line of the table), or Cargo table (table to be recreated).
3. ...no errors detected. The command to recreate data just returns a 200 OK.
4. Invoke runJobs.php. Observe that only 500 rows exist.

SovereignGFC (talk) 01:32, 2 June 2023 (UTC)

If you run cargoRecreateData.php from the command line on that table, does that work? Yaron Koren (talk) 13:09, 7 June 2023 (UTC)

Vertical list of names be grouped by the initial letters

Sorry if my question has already been asked here, but I have just started using Cargo. How could a vertical list of names (from Cargo databases) be grouped by the letters of the alphabet using a Cargo query? Lalo5555 (talk) 08:49, 27 May 2023 (UTC)

If you use the "category" format, it will provide a display that's hopefully close enough to what you are looking for. Yaron Koren (talk) 02:36, 29 May 2023 (UTC)

Export as Excel, Composer, What Did I Miss?

(This is preventing me from properly testing the 500-line recreate bug as I want to back up my big table beforehand!)

When I try to export as Excel, I get a 500 internal server error when I click on the link to download. Unfortunately, even with MediaWiki debug on, that's all I do get:

HTTP/1.1 500 Internal Server Error
Date: Wed, 31 May 2023 02:41:54 GMT
Server: Apache
X-Powered-By: PHP/7.4.33
X-Content-Type-Options: nosniff
X-Request-Id: ZHaz8pqI3nmiV89-O7ng8wAAAss
Strict-Transport-Security: max-age=63072000; includeSubDomains
X-Frame-Options: SAMEORIGIN
Content-Length: 0
Connection: close
Content-Type: text/html; charset=UTF-8

I have Composer installed at the same level of directory as MediaWiki and I confirm that phpoffice/phpspreadsheet is installed (1.19.0). I know it works because I can import spreadsheets just fine...but not export for some reason. SovereignGFC (talk) 02:46, 31 May 2023 (UTC)

The "excel" format code is very old, and it actually uses the PHPExcel library, not PhpSpreadsheet. Yaron Koren (talk) 13:04, 31 May 2023 (UTC)

Inspired by this discussion, I just added support for PhpSpreadsheet to the "excel" format. At least, in theory - I haven't actually tried running it. But there's a chance that it works! Yaron Koren (talk) 15:54, 1 June 2023 (UTC)

Can confirm this works!

I did the very proper and sophisticated thing of "copy entire single changed file and paste text into existing version" before running the query.

All 800+ items exported without issue. SovereignGFC (talk) 00:51, 2 June 2023 (UTC)

That's great to hear! Yaron Koren (talk) 02:44, 2 June 2023 (UTC)

One item of note: It defaults to the XLS extension, not XLSX. Excel 2021 complains the file doesn't match the extension and it may be a security issue.

I'm guessing that either specifying XLS format and keeping the XLS extension or changing it to use XLSX would make this nag go away. SovereignGFC (talk) 12:12, 5 June 2023 (UTC)

Ah, good point. XLSX seems to be the superior format, so I just changed the default filename to be "results.xlsx". Yaron Koren (talk) 15:03, 5 June 2023 (UTC)

Remove unwanted comma from results if fields are empty

I have this table with the fields ImageText and ImageFile being optional:

{{#cargo_store:_table=Countries}}

{{#cargo_declare:_table=Countries |Capital=String |Population=Integer |ImageText=Text |ImageFile=String }}

If ImageText and ImageFile have values, they display framed images with captions using this query:

{{#cargo_query:table=Countries|fields=CONCAT('[[File:',ImageFile,'|frame|',ImageText,']]')|default=}}

This works fine, but how do I remove the commas from the display when ImageText and ImageFile are empty? Tahoma403 (talk) 04:22, 7 June 2023 (UTC)

It could that just adding where=ImageFile IS NOT NULL AND ImageText IS NOT NULL to the #cargo_query call will fix the problem. Yaron Koren (talk) 13:10, 7 June 2023 (UTC)

That removed <p>, , , , , </p> from the top of the results, but in between each <div class="thumb tright">, I still get one <p>, </p> for some reason. Any idea? Tahoma403 (talk) 06:23, 8 June 2023 (UTC)

I think that's just due to MediaWiki's own display of images, though I could be wrong. Have you considered using the "gallery" format for this data, by the way, instead of assembling wikitext? It might be a nicer-looking output. Yaron Koren (talk) 12:32, 8 June 2023 (UTC)

Yes, that must be the reason then. I have looked at format=gallery before, but couldn't figure out where to put ImageFile in the query - would you mind showing where/how to put it in the fields= parameter to display the image and caption? Tahoma403 (talk) 13:09, 10 June 2023 (UTC)

Well, now that I think about it, to use "format=gallery", you would first need to change the type of "ImageFile" from String to File in the #cargo_declare call. Once that is done (and the table is recreated), a simple query like this might work: {{#cargo_query:table=Countries|fields=ImageFile|format=gallery|caption field=ImageText}}. Yaron Koren (talk) 13:13, 13 June 2023 (UTC)

Output as RSS (or Atom or whatever)?

I'm hoping to create RSS/whatever from the output of wikitext. My first thought was that Cargo might do that, so I checked the documentation pages, but didn't notice anything about it. So, I asked about it on the main MediaWiki support desk (here). One of the suggestions was that Cargo can do it. So I looked around here some more, but still haven't been able to find anything about it.

Can Cargo do this? If so, could you please point me towards info on how to make it do so? Thanks! -Rwv37 (talk) 03:50, 19 June 2023 (UTC)

@Rwv37: It's a new feature, and hasn't been included in a release yet. If you're running the master version of Cargo, then you can do format = rss (or atom) in a query, to get a link to an RSS feed. Sam Wilson 06:12, 19 June 2023 (UTC)

I meant |format=feed (the type can be specified with |feed type=atom). Sam Wilson 07:25, 19 June 2023 (UTC)

@Samwilson: Great - thank you! Rwv37 (talk) 22:04, 19 June 2023 (UTC)

This is great news. Thanks for working on it. Jonathan3 (talk) 13:52, 20 June 2023 (UTC)

Just for the benefit of anyone else who wants to try this out: It seems like you probably need to use MediaWiki 1.39 or higher. I'm using 1.38, and I get an error indicating that it can't find the class "MainConfigNames", which (it looks to me) was added in 1.39 -Rwv37 (talk) 04:13, 21 June 2023 (UTC)

@Rwv37: That's odd. MediaWiki\MainConfigNames was added in 1.38, so it should be working. Sam Wilson 04:51, 21 June 2023 (UTC)

@Samwilson, I see that that page says 1.38, but: I'm largely ignorant about the internals of MediaWiki, and maybe I'm misinterpreting something, but is it possible that's just a mistake? I ask for a few reasons:

(1) When I actually try it, I get an error saying it can't find that class.

(2) This page (for MainConfigNames.template) says 1.39+.

(3) Also on that page, there are links for its source code for various MW versions; if you click on the 1.39 one, you're brought to code, but if you click on 1.38 or earlier, you're brought to a "no such file" page.

(4) My 1.38 installation does not have any file of that name (or any *MainConfigNames* file, case insensitive).

-Rwv37 (talk) 19:25, 21 June 2023 (UTC)

@Rwv37: You're quite right! That file was introduced in Gerrit change 766849 and is not included in 1.38 (but only 1.39 and above). Here's a patch for Cargo: Gerrit change 932020 (@Yaron Koren could you have a look?), and another one for bumping the @since for MainConfigNames: Gerrit change 932023. Sam Wilson 22:38, 21 June 2023 (UTC)

@Samwilson Thanks! I'm not sure of something, though: If I check out the latest git master branch now (as per the installation instructions), will this patch be included? Or would it have to wait for @Yaron Koren to merge it in or something? Thanks again. -Rwv37 (talk) 04:00, 23 June 2023 (UTC)

@Rwv37: Yep, Yaron has already merged 932020 so if you pull the latest master now then it should be okay for 1.38. Sam Wilson 04:03, 23 June 2023 (UTC)

@Samwilson Just got a chance to give the latest a try with 1.38 - only a little bit, and I'll definitely be playing around with it more, but it seems to be working as far as I've checked! I have noticed one minor bit of seeming weirdness, but it may be due to how I'm using it. I'll eventually open up a new section on this page for it if I can't figure it out myself.

Thanks again! -Rwv37 (talk) 01:10, 29 June 2023 (UTC)

@Rwv37: Great! Let me know. I wonder if the weirdness is related to timezones… :-P Sam Wilson 01:31, 29 June 2023 (UTC)

RSS: Setting guid

Tracked in Phabricator
Task T342954

How is the guid set? It looks like it's just using the URL of the page (https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Cargo/+/888110/18/includes/formats/CargoFeedFormat.php#125) I seem to remember having a problem with the old "BulletFeed" extension that RSS readers wouldn't know which were new items. Sometimes my items were different but shared the same URL. I ended up editing my copy of it to specify that guid was the URL plus a URL parameter based on the pubDate. Jonathan3 (talk) 10:59, 21 June 2023 (UTC)

@Jonathan3: Yes, that's right, it's the URL of the page, but can be overridden by a query's url field. I think perhaps it should use the revision ID instead, as the default link? But that doesn't look as nice for consumers of the feed. I wondered about using oldid but then it might not link to the current version. Open to suggestions either way! :-) Sam Wilson 11:07, 21 June 2023 (UTC)

I think people are suspicious of URL parameters that aren't obvious (are they tracking? etc) so I used something clearly based on the date and time. The link and guid are the same - I can't remember why I didn't have the date/time only in the guid - maybe that would just have entailed more work on my part. I had to update the BulletFeed extension most times I upgraded MediaWiki. It bases its RSS feed on Cargo query results, so it will be good to cut out the middle man! I'll experiment with Cargo once I've upgraded to 1.38 and got confirmation that it works with that version :-)

The default you have is probably all right for most people. Especially as the url parameter which seems to set both guid and link can be modified in the Cargo call. That might be enough for me too. Jonathan3 (talk) 12:35, 21 June 2023 (UTC)

@Jonathan3: I've been wondering a bit about this, and think that perhaps the link should be the canonical page URL, and the guid should use the revision ID (e.g. …/index.php?oldid=123. That way, consumers know that the item is 'new' and will re-fetch it. Because often the publication date won't be updated even if the content is. Sam Wilson 01:30, 29 June 2023 (UTC)

That sounds good. In my case the RSS pubDate is saved as a Cargo date/time field, and is when the page was originally created. I can't remember why I did this but probably to get round a weakness in the original BulletFeed extension. How will Cargo set it in the RSS feed output format - page creation date or something else? Jonathan3 (talk) 11:09, 29 June 2023 (UTC)

Thinking about it, how would the Cargo query know the revision ID to use? Maybe it would be good to allow the user (Cargo query writer) to set the GUID as part of the query. Jonathan3 (talk) 13:55, 21 July 2023 (UTC)

@Samwilson: "Yes, that's right, it's the URL of the page, but can be overridden by a query's url field." Looking at the code, and from experimenting, it seems that the url element can be overridden by the query's url field, but the guid element cannot: it is always taken from the canonical url of the page. It would be better if both were the same, or at least if there were a way of setting the guid in the query. Jonathan3 (talk) 18:23, 24 July 2023 (UTC)

Looking again, the guid seems to be based on a URL based in turn on the title element, even if that title element has been modified by a CONCAT statement in the Cargo query, i.e. the guid isn't necessarily a working URL. Another reason to set it as equal to the url or be overridden in the query :-) Jonathan3 (talk) 18:35, 24 July 2023 (UTC)

I've added a Phabricator task about this: RSS: Set guid from Cargo query to ensure it is unique. Jonathan3 (talk) 10:40, 28 July 2023 (UTC)

Duplicate results in Special:Drilldown

Tracked in Phabricator
Task T317306

Have you had a chance to look at this yet (Extension_talk:Cargo/Archive_January_to_April_2023#Duplicate_results_in_Special:Drilldown)?

It would be great if you could fix it. I'd be happy to try to help. I came up with an idea on the Phabricator task (task T317306) but couldn't get any further than that with it. Basically I wondered whether a GROUP BY could be added to the Drilldown page's query.

Thanks. Jonathan3 (talk) 12:43, 21 June 2023 (UTC)

Sorry for the long delay in responding to this. Yes, I'm aware of the problem, and yes, adding something like "GROUP BY" or "DISTINCT" is probably the right solution. My initial thought was that, if there are multiple items, i.e. template calls, on a page that match the current set of filters, it could be helpful for users to see that there are multiple matches. But that's probably silly. Yaron Koren (talk) 21:11, 22 June 2023 (UTC)

Yes, I think your second thought is correct :-) At most it might be useful to have a number in brackets after the page name to indicate how many times it meets the various filter values, but I wouldn't find that useful myself.

Where does the MySQL query used on the Drilldown page come from? I could experiment with changing it. Jonathan3 (talk) 21:16, 26 June 2023 (UTC)

Page name search on Drilldown page

There is a "Full text" search available, but it's for the contents of the pages rather than the page names. Is it possible to add a page name search? Thanks. Jonathan3 (talk) 21:14, 26 June 2023 (UTC)

Hide some tables from Drilldown page

I've worked out how to do this with nth-child.

But it would be nice if each table tab header had a unique css id. Could you possibly add this?

Thanks. Jonathan3 (talk) 21:57, 27 June 2023 (UTC)

[SOLVED] Remove "Choose a table:" from Drilldown page

This comes from (cargo-drilldown-choosetable)(colon-separator).

I can blank MediaWiki:cargo-drilldown-choosetable but don't want to do that with the colon-separator in case it's used elsewhere in the site.

Is CSS the only way round this? Thanks. Jonathan3 (talk) 20:27, 28 June 2023 (UTC)

Probably. Yaron Koren (talk) 00:12, 4 July 2023 (UTC)

Cheers. I used #tableTabsHeader { display: none; } Jonathan3 (talk) 00:26, 4 July 2023 (UTC)

"Order by" in Drilldown tabs?

I've tried this but it doesn't work. Should it?

_drilldownTabs=Table (format=table;fields=field1,field2,Date;order by Date DESC) Jonathan3 (talk) 20:40, 28 June 2023 (UTC)

I'd like to show the results in date order (newest first). Jonathan3 (talk) 20:45, 28 June 2023 (UTC)

I think ORDER BY wouldn't work because the tab display code will rely on whatever rows are given it by the main Drilldown code. I can get it to order by Date by putting Date as the first field. But then it's ordered Oldest first, which is the opposite of what I want :-) Is there any way round that maybe? Jonathan3 (talk) 20:54, 28 June 2023 (UTC)

Custom text at bottom of Special:Drilldown

Would you consider adding a MediaWiki: message to go at the bottom of the drilldown page?

After this part: (viewprevnext: (prevn: 100), (nextn: 100), 20(pipe-separator)50(pipe-separator)100(pipe-separator)250(pipe-separator)500)

Thanks. Jonathan3 (talk) 13:54, 29 June 2023 (UTC)

Database Error: Unknown column 'tl_namespace' in 'where clause'

I was trying out both SemanticMediaWiki and Cargo on my wiki. After trying each one I removed them. I decided to go with Cargo and tried to reinstall it again. It seemed like it went well until I tried to create a data table.

The error is saying "Did you forget to run your application's database schema updater after upgrading or after adding a new extension?" which I have already ran php update.php.

The full error is:

Error 1054: Unknown column 'tl_namespace' in 'where clause'

Function: SpecialCargoRecreateData::getNumPagesThatCallTemplate

Query: SELECT COUNT(*) AS total FROM `page`,`templatelinks` WHERE (tl_from=page_id) AND tl_namespace = 10 AND tl_title = 'Person'

Any suggestions on how to overcome this error? 70.160.223.43 22:43, 3 July 2023 (UTC)

What version of MediaWiki and Page Forms are you running? Yaron Koren (talk) 00:10, 4 July 2023 (UTC)

I typed my reply before reading yours! :-) In my head I still was on the Support Desk page. If I'd realised I'd have left the replying to you. Jonathan3 (talk) 00:19, 4 July 2023 (UTC)

Thanks, but I just got it back to work. I'm using MW 1.39.3 and when I downloaded Cargo I used Special:ExtensionDistributor/Cargo for 1.39 (but I think that version is what was messing it up). When I removed everything and downloaded from the link on Extension:Cargo/Download_and_installation the error was gone.

It seems the tables that first version of Cargo had was for MW 1.38 Manual:Templatelinks_table. 70.160.223.43 00:19, 4 July 2023 (UTC)

While I'm here, I should predict that Yaron is going to say that he doesn't recommend downloading from Extension Distributor and you should just run the latest code of his extensions :-) Jonathan3 (talk) 00:22, 4 July 2023 (UTC)

You predicted well. :) Yaron Koren (talk) 00:46, 4 July 2023 (UTC)

Something similar here: task T315635.

This seems to be the relevant Cargo code: https://github.com/wikimedia/mediawiki-extensions-Cargo/blob/master/includes/specials/SpecialCargoRecreateData.php#L168

What versions of MW and Cargo are you using? Jonathan3 (talk) 00:17, 4 July 2023 (UTC)

API Get Page Values

Using MediaWiki's API, I'd like to look up the page values for a specific page. I'm looking at the list of Cargo's APIs but I'm not finding which one to use. 70.160.223.43 01:48, 4 July 2023 (UTC)

I don't think there's a way to get all the Cargo values, across all of the tables, for a page via the API, unfortunately. Yaron Koren (talk) 14:52, 5 July 2023 (UTC)

Thank you very much. For anyone that comes in after looking for the same type of setup, I ended up having my script directly reading from the database tables. I know its probably not good practice or recommended for the vast majority, but it seems like it will work perfectly for me. 70.160.223.43 00:51, 6 July 2023 (UTC)

Duplicate entry 'XXXX' for key 'PRIMARY'

I'm getting a lot of duplicate errors for for my tables. I'm running MW 1.39.3 with Cargo 3.4.2.

To troubleshoot I've set $wgJobRunRate = 0; and I don't have any cronjobs running runJobs.php.

Each time I've gotten the error I've deleted the table using Special:CargoTables and then I recreated the table on Template:XXX.

To populate the table I'm using php cargoRecreateData.php.

Sometimes it's able to finish successfully, or sometimes it gets a fatal error for duplicate entry.

Is there something I can do to prevent this? How should I proceed each time this happens? And is it possible to change the code to an INSERT IGNORE instead of an INSERT command to prevent having that error be fatal? 70.160.223.43 21:15, 4 July 2023 (UTC)

There was actually one change to the Cargo code, in May, after the release of version 3.4.2, that might help with this problem - which reminds me that I should release a new version of Cargo. If you are using the actual version 3.4.2, from April, I would recommend upgrading to the very latest Cargo code, to see if that fixes the issue. If that doesn't work (or if you were already using more recent code), another option to try is to add the following to LocalSettings.php: $wgCargoIgnoreBacklinks = true; . Yaron Koren (talk) 14:50, 5 July 2023 (UTC)

I used the zip link on Extension:Cargo/Download_and_installation which I downloaded a few days ago. Is there a way I can check to see if I have the correct version?

For $wgCargoIgnoreBacklinks, would there be any downsides or things to be concerned of if I set this to true? 70.160.223.43 00:55, 6 July 2023 (UTC)

That's the "old" version, i.e. the true 3.4.2. But that's alright - that setting will hopefully get rid of the error also. There are no known downsides to it, other than Cargo queries not being immediately refreshed when there is a change to their data. Yaron Koren (talk) 12:59, 6 July 2023 (UTC)

Recreate table internal error, missing function "getLinkTargetIId"?

I just tried to recreate a table for the first time in quite a while. I get an error saying:

Call to undefined method MediaWiki\Linker\LinkTargetStore::getLinkTargetId()

Things seem to be working fine otherwise (for example, I can use Cargo to stick stuff in the database and retrieve stuff from it).

I am using MediaWiki 1.38, and the master branch of Cargo. Initially I had been using the master revision from about a week ago (to get the new RSS feature), and I got this error with it. I then did a git pull to get to the latest master revision (just a minute or two ago), and I'm still getting the error. Thanks in advance for any help.

Full error output:

[ZKekih2gxzJJq2a2GsaZ0AAAAAk] /Wikis/Foodclubipedia/0001/index.php?title=Template:What%27s_new/Cargo&action=recreatedata Error: Call to undefined method MediaWiki\Linker\LinkTargetStore::getLinkTargetId()

Backtrace:

from /home/bob/foodclub.us/Wikis/Foodclubipedia/0001/extensions/Cargo/includes/specials/SpecialCargoRecreateData.php(173)

1. 0 /home/bob/foodclub.us/Wikis/Foodclubipedia/0001/extensions/Cargo/includes/specials/SpecialCargoRecreateData.php(84): SpecialCargoRecreateData->getNumPagesThatCallTemplate(Wikimedia\Rdbms\MaintainableDBConnRef, Title)
2. 1 /home/bob/foodclub.us/Wikis/Foodclubipedia/0001/extensions/Cargo/includes/CargoRecreateDataAction.php(43): SpecialCargoRecreateData->execute()
3. 2 /home/bob/foodclub.us/Wikis/Foodclubipedia/0001/includes/MediaWiki.php(543): CargoRecreateDataAction->show()
4. 3 /home/bob/foodclub.us/Wikis/Foodclubipedia/0001/includes/MediaWiki.php(321): MediaWiki->performAction(Article, Title)
5. 4 /home/bob/foodclub.us/Wikis/Foodclubipedia/0001/includes/MediaWiki.php(912): MediaWiki->performRequest()
6. 5 /home/bob/foodclub.us/Wikis/Foodclubipedia/0001/includes/MediaWiki.php(563): MediaWiki->main()
7. 6 /home/bob/foodclub.us/Wikis/Foodclubipedia/0001/index.php(53): MediaWiki->run()
8. 7 /home/bob/foodclub.us/Wikis/Foodclubipedia/0001/index.php(46): wfIndexMain()
9. 8 {main}

Rwv37 (talk) 05:56, 7 July 2023 (UTC)

You must be using an alpha version of MW 1.38. Anyway, I just checked in a fix to Cargo to handle that MW code. Yaron Koren (talk) 15:58, 7 July 2023 (UTC)

@Yaron Koren, I don't think I'm using an alpha version - I'm not sure that I'd even know how to get one without investigating first, and I don't think I've investigated. The version I'm using claims to be 1.38.2.

Anyway, thank you! I updated to the latest Cargo master, and it's working fine now. Also, I see that 1.38 is now "obsolete" as of a week ago, so I'll look into upgrading to 1.39 (LTS) and hopefully not have to bother you about 1.38-related stuff in the future. Thanks again! - Rwv37 (talk) 20:50, 7 July 2023 (UTC)

Oh, interesting. In any case, Cargo still supports MediaWiki versions going back to 1.35, so I'm glad the fix worked. Yaron Koren (talk) 21:08, 7 July 2023 (UTC)

RSS: Feed icon?

In the new RSS feed stuff, is there a way to specify the feed's icon? Or even just to get it to be the same as the wiki's icon? For me, it seems to be just defaulting to the website's icon (which is not the same as the wiki's icon). Rwv37 (talk) 06:15, 7 July 2023 (UTC)

CC @Samwilson: for this and the below questions. Yaron Koren (talk) 16:00, 7 July 2023 (UTC)

@Rwv37: This sounds like a good idea. The MediaWiki feed classes don't appear to support image (RSS) or logo (Atom), so we'd have to add support for those there. It'd probably be good for all wiki feeds to default to the wiki's logo anyway (although I'd have thought that someone would've suggested that in the last 20 years, so maybe there's a reason against it). Atom says that "The image SHOULD have an aspect ratio of 2 (horizontal) to 1 (vertical)" so the new 'wordmark' variant in $wgLogos could be a suitable default. Sam Wilson 02:06, 9 July 2023 (UTC)

RSS: Wikitext for "link text"?

It would be nice to allow markup in the "link text" field. For example, you could include a standard RSS image. Rwv37 (talk) 06:27, 7 July 2023 (UTC)

@Rwv37: I'm not sure. The other usages of that parameter don't support wikitext, nor do others such as |more results text=. Perhaps it'd be better to set up a URL rewrite for the feed, and then just use normal wikitext for linking to it? e.g. [[File:Feed-icon.svg.png|25px|link=https://wiki.example.org/news.rss]] Sam Wilson 06:57, 10 July 2023 (UTC)

RSS: Publication date

The docs say "It'll use the first date field found for the publication date". I'm including a Date field as one of the fields, but the feed is just assigning what looks to me like Unix-time-zero to every feed entry's "pubDate". Rwv37 (talk) 06:33, 7 July 2023 (UTC)

This won't necessarily help you, but I'm doing the same and it works fine for me. What's your query and what does the RSS look like? Jonathan3 (talk) 18:28, 24 July 2023 (UTC)

Removing Wikitext From Field

One of the fields I'm collecting, basically a summary, has wikitext in it. The summary on the wiki should have wikitext, so that's fine, but when I extract that text I'd like to just have plain text. I'd also like to NOT maintain two almost identical summaries side by side on the wiki where one has wikitext and one doesn't.

Does Cargo have any built in options or are there any recommended methods for removing wikitext? 70.160.223.43 12:23, 8 July 2023 (UTC)

I don't know any standard way - in core MediaWiki, Cargo or elsewhere - to turn formatted wikitext into plain text (which would presumably include removing links, etc.). If you want to do this, you may have to create a Lua module for it or that sort of thing. Yaron Koren (talk) 14:19, 10 July 2023 (UTC)

Thanks 70.160.223.43 18:59, 11 July 2023 (UTC)

I use Extension:Popups. I can't remember how I set it up (I'd rather if it showed properly parsed wikitext!) but the popup just shows the plain text of a page. For instance, if the page begins "'''This page''' has bold text" (so usually displays as "This page has bold text"), then the popup will just show "This page has bold text". You might be able to see how the extension does it. Jonathan3 (talk) 13:31, 21 July 2023 (UTC)

RSS: Short URL not being used

(tagging @Samwilson since @Yaron Koren seems to be doing so for RSS-related stuff... if this is not appropriate for me to do, sorry, and please let me know)

I have my wiki set up to use short URLs, like https://foodclub.us/Foodclubipedia/What%27s_new . Cargo's RSS stuff seems to use long URLs (at least by default), like https://foodclub.us/Wikis/Foodclubipedia/0001/index.php?title=Special:CargoExport&tables=WhatsNew&fields=_pageName%2C+OriginalDate%2C+BriefSummary&limit=100&format=feed&feed+type=rss&feed+title=What%27s+New+on+Foodclubipedia&feed+description=The+greatest+Foodclubipedian+What%27s+New+syndication+feed+of+all+time%21. Is there a way to get it to use a short URL instead? Thanks. Rwv37 (talk) 19:27, 8 July 2023 (UTC)

In case it's not clear, I think this is more than just "use a nicer-looking URL". The "/0001/" in the URL is related to a system I've set up that would make it relatively easy to create and test a new version (presumably "/0002/") without impacting users at all, and (when I'm satisfied with my testing) to instantly migrate to the new version without users even noticing. Rwv37 (talk) 19:38, 8 July 2023 (UTC)

@Rwv37: I've found that issue as well, including with other Cargo export formats. The way I've dealt with it is via a web server URL rewrite. For example, if you use Apache you could use something like this:

RewriteRule ^/Foodclubipedia/What's_new\.rss$ "/Foodclubipedia/0001/index.php?title=Special:CargoExport&tables=WhatsNew&fields=_pageName%2C+OriginalDate%2C+BriefSummary&limit=100&format=feed&feed+type=rss&feed+title=What%27s+New+on+Foodclubipedia&feed+description=The+greatest+Foodclubipedian+What%27s+New+syndication+feed+of+all+time%21" [L]

Then you could change that any time you want to change the parameters of the feed. You could even add variables in there so it could change based on the input URL (e.g. for tags or categories). Sam Wilson 01:47, 9 July 2023 (UTC)

Cool, thanks. I should've thought of that - I'm already using mod_rewrite extensively (including but not only for wikis).

Thanks again! - Rwv37 (talk) 01:55, 9 July 2023 (UTC)

RSS: Fully canonical URL for feeds?

Somewhat related to the "RSS: Short URL not being used" question I added above (but I think would be even better):

Can it somehow get all the stuff like "tables=", "fields=", etc. not from the URL, but from configuration? So, for example, something like https://example.com/Examplepedia/WhatsNewFeed, with "WhatsNewFeed" looked up by the host and resolved into a particular RSS feed?

That way, the end user would not be impacted by internal changes that they shouldn't care about (like backend DB layout). Thanks.

(CC @Samwilson)

- Rwv37 (talk) 19:34, 8 July 2023 (UTC)

Would also be nice if the extension could be asked for the canonical URL, so that you could easily link to it from multiple pages/places instead of defining the feed all over the place. Right now I'm doing that via a template, but that can't be used in all situations (at least not in the potentially dumb way I've built it). - Rwv37 (talk) 20:00, 8 July 2023 (UTC)

Could dropdown on Drilldown page include the filter values that are displayed as links?

I wonder whether it would be good for the dropdown to search through every filter value, not just the ones that aren't listed as links. It might make it easier to find things at a glance. Is this possible? Jonathan3 (talk) 08:56, 20 July 2023 (UTC)

RSS: Does "feed" export display format work with MW1.35?

Tracked in Phabricator
Task T342961

When I set up a simple query it shows the "View RSS feed" link, but clicking that gives a 500 error. There was a discussion here about whether an aspect of this worked on 1.38 or just 1.39. Maybe the feed format requires 1.38? @Samwilson I hope you don't mind me pinging you here :-) Jonathan3 (talk) 13:22, 21 July 2023 (UTC)

@Jonathan3: I thought we fixed the issue with that? Can you find out what the actual error message is? Maybe it's something different. Sam Wilson 13:25, 21 July 2023 (UTC)

Thanks for the quick reply. I know how to turn on debugging in MediaWiki but it's some sort of server problem (the 500 error) and I haven't been able to delve any deeper than that yet. I just wondered whether it was known not to work with 1.35, in which case I would have done nothing before upgrading. Jonathan3 (talk) 13:34, 21 July 2023 (UTC)

format=csv works so that might narrow it down a bit. Is there a basic Cargo query for RSS that should work that I can try? Jonathan3 (talk) 13:36, 21 July 2023 (UTC)

I wonder why it's not working for me. I can't see anything relevant in apache2/access.log or apache2/error.log, though I'm not an expert on this sort of thing at all. Where else should I look? Jonathan3 (talk) 10:38, 24 July 2023 (UTC)

It breaks at this line: https://github.com/wikimedia/mediawiki-extensions-Cargo/blob/master/includes/formats/CargoFeedFormat.php#L99 $contentRenderer = MediaWikiServices::getInstance()->getContentRenderer(); Jonathan3 (talk) 10:49, 24 July 2023 (UTC)

@Samwilson: When I comment out that line (and ensure that the query has a "description" field so that it is not needed later) the feed seems to work. No 500 error anyway! I'm on MW 1.35.8 and can provide further details if you need them. Jonathan3 (talk) 11:03, 24 July 2023 (UTC)

I've asked on the Support Desk to get a wider audience: Topic:Xmka656mt9uyskph. Jonathan3 (talk) 14:38, 25 July 2023 (UTC)

Please find my answer at https://www.mediawiki.org/w/index.php?title=Topic:Xmka656mt9uyskph&topic_showPostId=xmkc4g47j31tfm17#flow-post-xmkc4g47j31tfm17 - Quick answer: Upgrade to MW 1.39 --Osnard (talk) 15:14, 25 July 2023 (UTC)

Thank you very much Osnard. Sam, would you be willing to change the code to be compatible with MW1.35? Thanks in anticipation. Jonathan3 (talk) 20:36, 25 July 2023 (UTC)

[SOLVED] RSS: Compound query?

I haven't been able to experiment yet but think I might need to use a compound query to get the effect I need... will this be possible? Thanks. Jonathan3 (talk) 13:47, 21 July 2023 (UTC)

@Jonathan3: The feed format is just like other output formats, and will work with whatever results you throw at it. It has no special handling for multiple tables (if that's what you mean by compound query?), and needs none I think. Sam Wilson 23:24, 21 July 2023 (UTC)

[SOLVED] RSS: Title of each item

How can the title of each item be set? I think at the minute it's just using _pageTitle. I'd like to be able to use some text which includes the page title. Jonathan3 (talk) 13:50, 21 July 2023 (UTC)

@Jonathan3: If you change the field that has the name _pageName then it should use that, e.g. CONCAT('Post: ', short_description)=_pageName. Sam Wilson 23:22, 21 July 2023 (UTC)

That works, thanks :-) Jonathan3 (talk) 18:25, 24 July 2023 (UTC)

#cargo_query Output Variables

Currently I use #cargo_query with format template output to create some basic wikitext. Something like "[[Categeory:{{{1|}}} X]]" or like "[[Y {{{1|}}}]]"

Is there some #cargo_query output where I can output this data without having to create and manage separate templates? 70.160.223.43 17:05, 22 July 2023 (UTC)

You can use CONCAT() within the "fields=" value, to assemble the strings within the query instead of within a display template. Yaron Koren (talk) 14:06, 24 July 2023 (UTC)

Thank you very much.

I'm using |fields=CONCAT("[[Category:", Role.library, "]][[Category:", Role.position, "]]"). I don't have format specified, so with my understanding, it will use list as concatenating everything together makes it be one field.

In my output the category wikitext is working properly, but it's also showing the text in the page as well. Appearing like if there was a duplication of code and the first set was wrapped in nowiki tags. Is there something I need to change? 70.160.223.43 21:09, 11 August 2023 (UTC)

RSS: Automatically find "link" from "description" field

Tracked in Phabricator
Task T342957

I'd like it if the "link" could be generated automatically from the first link to appear within the "description" field. Is this an option that you'd be willing to add? If not, I'll have a think and likely can find a way round it (maybe by manually storing the link as a separate Cargo field). (@Samwilson - does being pinged here help or annoy you more?) Thanks. Jonathan3 (talk) 14:10, 24 July 2023 (UTC)

RSS: Parse wikitext in description field

Tracked in Phabricator
Task T342955

@Samwilson: If a link appears within the wikitext field, the RSS description tag contains the wikitext (e.g. [http://www.example.com Example]. I'd like it to contain HTML (e.g. <a href="http://www.example.com">Example</a>. Is that possible? Thanks :-) Jonathan3 (talk) 14:13, 24 July 2023 (UTC)

I think this is the one thing that I definitely can't think of a workaround for, before I start using Cargo for RSS. I think it would make sense for the description to include HTML instead of wikitext, as generally the RSS feed would be displayed on an external website rather than on another wiki. And I think a wiki would be able to display the HTML anyway (if not, perhaps HTML vs wikitext could be an option in the Cargo query). Jonathan3 (talk) 14:50, 25 July 2023 (UTC)

@Jonathan3: Sounds possible, I'll see what I can do. Feel free to open Phabricator tasks for these things, by the way, so we can keep track of them. :-) Sam Wilson 09:01, 28 July 2023 (UTC)

Cool. Will do! Jonathan3 (talk) 10:28, 28 July 2023 (UTC)

How to see errors from failed cargo_store commands?

I've had it multiple times now that a template, which declares a cargo table and stores entries in it, fails to add entries to the table. When this happens, where can one see the SQL error that causes the INSERT command to fail? TaylanKammer (talk) 18:30, 25 July 2023 (UTC)

If you recreate the table via the command line it'll show errors.

I think you can set up MediaWiki to log job queue errors. Jonathan3 (talk) 18:40, 25 July 2023 (UTC)

Documentation issue

On MW 1.39.4, the suggested fix for pages with ' in titles as seen on the Common problems:

If the "where" clause contains something like _pageName = 'Cargo/Archive May to July 2023', the query will break for pages that contain a single quote/apostrophe (') in their name. There are a few potential fixes for this: the easiest is to change the query to _pageID = 1837568, which should always work. Another option is to change it to _pageName='{{#replace:{{PAGENAME}}|'|\'}}'. (You could also consider replacing the single quotes around Cargo/Archive May to July 2023 with double quotes, but single quotes are generally preferred for SQL literals, plus this formulation will itself break if the page name contains double quotes.)

The above suggestion {{#replace:{{PAGENAME}}|'|\'}} doesn't work on vanilla MediaWiki 1.39.4 (and thus not with Cargo either), possibly because of what the ParserFunctions page states that:

Some magic keywords or parser functions of MediaWiki (such as {{PAGENAME}} and similar) are known to return strings that are needlessly HTML-encoded, even if their own input parameter was not HTML-encoded.

Doing the suggested:

• {{#replace:{{PAGENAME}}|'|\'}} (where {{PAGENAME}} is Some'name) still gives: Some'name.

However, the following works (using titleparts to return non-html-encoded text):

• {{#replace:{{#titleparts:{{PAGENAME}}}}|'|\'}} (where {{PAGENAME}} is Some'name) gives: Some\'name (as desired, doesn't break the query).

Next step, testing if the latter works in a MATCHES statement with multiple criteria (two different sets of quote marks). Ideally, MATCHES should distinguish between a page name's ' and its own, but it can't. That led me to the docs, which led me astray with that nonworking example above, found when I started debugging each line in my statement. I'm not sure if this is a MW version thing or not, if I was, I'd update the docs accordingly. TiltedCerebellum (talk) 20:10, 29 July 2023 (UTC)

Update: Using ' as suggested also doesn't work in the WHERE command, using MATCHES with multiple terms (which uses two different quote types); it gave me "Error: unclosed string literal.". Nor did this work:

• {{#replace:{{#titleparts:{{PAGENAME}} }}|'|\'}} ("Error 1064: You have an error in your SQL syntax;").

But this did work:

• {{#replace:{{#titleparts:{{PAGENAME}} }}|'|\’}}.
  • Interestingly, when I query against either ' or ’ for pagename, the same results are returned for either (Cargo sees them as interchangeable), which actually works well for us, as it provides a workaround (in case it helps someone else). TiltedCerebellum (talk) 20:48, 29 July 2023 (UTC)

I've not read through all this yet, but very many of my pages have ' characters in their titles. I just use " characters in the Cargo query and it works all right, e.g. |where=_pageName="{{{1}}}" or |where=_pageName="{{PAGENAME}}". I only have to do anything fancy for page names with both " and ' characters, but I avoid that now! I should add that I'm on MW1.35 but am about to upgrade to 1.39 so am interested in whether the version is relevant here. Jonathan3 (talk) 22:34, 29 July 2023 (UTC)

Thanks for the thought, it doesn't apply. ;-) -- Probably specific to Cargo's MATCHES handling, when already using two quote types (if not using, probably won't run into). I added a bit above to clarify.

With MATCHES (used to search _pageData._fullText columns), when multiple matches are needed, it already uses both ' and " characters... e.g., |where=_pageData._fullText MATCHES '"item1" "item2" "item3"' (this works, spaces are equivilent to OR, though this is where special characters in page names break stuff). Probably could be rewritten in a long chain (though manual states it follows MATCHES...AGAINST function, which allows/demonstrates the former). Next, vanilla MediaWiki (1.39.4) cannot do the suggested {{#replace:}} workaround for similar cases in WHERE commands:

If the "where" clause contains something like _pageName = 'Cargo/Archive May to July 2023', the query will break for pages that contain a single quote/apostrophe (') in their name. There are a few potential fixes for this: the easiest is to change the query to _pageID = 1837568, which should always work. Another option is to change it to _pageName='{{#replace:{{PAGENAME}}|'|\'}}'. (You could also consider replacing the single quotes around Cargo/Archive May to July 2023 with double quotes, but single quotes are generally preferred for SQL literals, plus this formulation will itself break if the page name contains double quotes.)

• The suggested {{#replace:{{PAGENAME}}|'|\'}} (doesn't work at all in MW 1.39.4 for {{PAGENAME}} with ' in them--does nothing, so statement still breaks).
• {{#replace:{{#titleparts:{{PAGENAME}} }}|'|\’}} does work on a page name like "Li'l Angel" (see ParserFunctions quote above, where "sometimes" {{PAGENAME}} is "needlessly HTML encoded"), titleparts must be used to convert to non-HTML encoded--that notice has been there many versions, I suspect the suggested workaround either stopped working, or never worked.

- Forgot to add:

• And very oddly, this doesn't work either: {{#replace:{{#titleparts:{{PAGENAME}} }}|'|\'}}. Seems it needs the different quote symbol, and returns the same results when searched against despite different symbol used. No idea if intended ¯\_(ツ)_/¯ but greatful it works! TiltedCerebellum (talk) 02:04, 30 July 2023 (UTC)

Using "icon" paremeter with leaflet

Hello, I am trying to create a map with different coloured icons and am using the "icon" parameter. Unfortunately I haven't been able to get this to work and using custom icons leads to them appearing in the wrong location even if the icon file is the exact same size as the default blue icon. Please see here. Any ideas how to resolve this? Garuda3 (talk) 21:19, 31 July 2023 (UTC)

It would appear that the following CSS resolves the issue:

.leaflet-marker-icon {

margin-left: -12px;

margin-top: -41px;

width: 25px;

height: 41px;

transform: translate3d(615px, -247px, 0px);

z-index: -247;

opacity: 1;

outline-style: none;

}

Garuda3 (talk) 21:39, 31 July 2023 (UTC)