Extension talk:External Data/Archive 2009

External data showing non-cached data but semantic data incorrect

I'm pulling data from twitter and marking the data using semantic mediawiki, and the external data is updating, but the semantic properties aren't updating. They aren't updating even when I hit the refresh tab. See:

http://ec2-174-129-9-164.compute-1.amazonaws.com/wiki/Twitter_searches

http://ec2-174-129-9-164.compute-1.amazonaws.com/wiki/Adeletiblier

http://ec2-174-129-9-164.compute-1.amazonaws.com/wiki/Special:Browse/Adeletiblier

Any idea why this is going on?
--Ryan lane 03:11, 17 July 2009 (UTC)

Btw, if I actually modify the page, it updates the semantic data. Simple purges don't work though.
--Ryan lane 03:15, 17 July 2009 (UTC)

Yes, that's a definite problem when using External Data in conjunction with SMW - the data, once retrieved, is stored in the SMW tables, and after that it becomes hard to update it. Refreshing pages won't work, because that doesn't lead to any database updates. At the moment, I think the best solution (and the one that some people already use) is to create a cron job to run the SMW script "SMW_refreshData.php" (which does update the database) at a regular interval, like once a day. I should add something to the extension page about that...

Ah, this solves my problem, thanks!
--Ryan lane 13:51, 17 July 2009 (UTC)

By the way, your site's a great demonstration of the possibilities of External Data and SMW together - I might end up using it for demo purposes, assuming it stays up... Yaron Koren 13:44, 17 July 2009 (UTC)

Amazon EC2 is expensive ;). I put it up for demo purposes for a talk I'm giving at BarCampNOLA on community software development where SMW is the backend, SF is the frontend, templates are business logic, and Plotters and Gadgets are user writable javascript. It is unlikely I'll keep the site up unless I find some people who want to co-host (and co-pay). At some point I'll probably get a VPS. I have the database and wiki files saved, so I'll let you know the new site when I bring it up.
--Ryan lane 13:51, 17 July 2009 (UTC)

Ah, okay. I'm glad to hear that you're spreading the word about the "extended SMW" system. By the way, you can host a wiki that uses all those extensions for free on Referata. Yaron Koren 14:19, 17 July 2009 (UTC)

Allowing non-REST APIs?

I'm sure I know the answer to this, but would it be possible for this to support non-REST APIs in the future? Something like SOAP or XML-RPC?
--Ryan lane 13:57, 17 July 2009 (UTC)

I don't see why it wouldn't be possible... it just needs to be implemented, including coming up with a good way to express all the data being sent, via a parser function call (presumably something like #get_soap_data, etc.) Yaron Koren 14:21, 17 July 2009 (UTC)

Awesome. I may take a crack at this at some point in the future...
--Ryan lane 17:37, 17 July 2009 (UTC)

Error when running SMW_refreshData.php

I'm getting the following error when I run SMW_refreshData.php:

Fatal error: Call to a member function getText() on a non-object in /var/www/w/extensions/ExternalData/ED_ParserFunctions.php on line 20

Which is: $cur_page_name = $wgTitle->getText();

Do I have something misconfigured?
--Ryan lane 20:39, 17 July 2009 (UTC)

Oh yes, sorry about that - this has come up as an issue before. There's a fix that needs to be applied to Semantic MediaWiki to get this working. In the file "/includes/storage/SMW_SQLStore2.php", under line 1278 ("foreach ($titles as $title) {"), you should add the following two lines:

                        global $wgTitle;
                        $wgTitle = $title;

I should add that to the documentation as well; and of course add the patch to SMW. Yaron Koren 15:30, 18 July 2009 (UTC)

Cool. That works thanks!
--Ryan lane 03:10, 19 July 2009 (UTC)

Limit #get_external_data to specific URLs

For security reasons, it would be nice to limit the external data to trusted sites only. Would this be possible? If I gave a patch for this, would you accept?
--Ryan lane 19:34, 27 July 2009 (UTC)

Hm, that's a good idea - how about an "$edgAllowExternalDataFrom" variable, that holds an array of domain names, akin to MediaWiki's $wgAllowExternalImagesFrom setting? I'd definitely accept that. Presumably, if the value were null, all URLs would be accepted. Yaron Koren 19:42, 27 July 2009 (UTC)

That sounds perfect to me.
--Ryan lane 20:05, 27 July 2009 (UTC)

Internationalization

Looks like the error messages are echoed directly in english without i18n support. Can I send you a patch adding support?

Btw, I have commit access, do you prefer that I send patches, or commit and let you revert things that don't fit properly? Either way I'll ask first about changes I'd like to make.
--Ryan lane 20:55, 27 July 2009 (UTC)

Yeah, that's true, I never translated all the error messages; which would be a good idea. If you have commit access, I think it's definitely better if you check in changes directly; although it's good of you to ask first (most people don't bother :) ). Yaron Koren 22:39, 27 July 2009 (UTC)

Nothing Displays

When I use the example on your site I get nothing displaying.

I tried running the SMW_refreshdata.php and had to apply the above patch, but I now get the following error:

PHP Notice:  Undefined offset:  1 in /opt/apps/itwiki/itwiki/extensions/ExternalData/ED_ParserFunctions.php on line 56

I have also tried accessing a json source and I get the following string displayed on the page:

?UNIQb8b7293751eb2cb-pre-00000000-QINU?

My guess is that there's an error in your #get_external_data call - all parameters after the first two need to contain either '=' or '=='. You've definitely uncovered a bug, though - the error-handling should be more graceful. The second problem sounds bad, too - maybe it's caused by the same error? I hope so. Yaron Koren 13:42, 28 August 2009 (UTC)

I upgraded the version of SemanticMediaWiki to 1.4.3 from 1.4.2 (mediawiki 1.14), it works with json type external data but not with the csv germany example. Is there a page with tested versions on dependent apps i can check? -JS 12:42, 31 August 2009 (UTC)

Sorry, what do you mean by tested versions and dependent apps? The Discourse DB Germany page is, of course, a demo page... Yaron Koren 19:57, 31 August 2009 (UTC)

eg. for version x of external data, the following versions of mediawiki, semanticmediawiki etc are needed/have been tested. I also had to add JSON support for php on my server. -JS 20:38, 31 August 2009 (UTC)

I see. No, there isn't... the CSV format is a special case, since in SMW 1.4.3 that format began to be printed out with a header row; so the format in #get_external_data should be 'csv with header' instead of 'csv'. Yaron Koren 21:53, 31 August 2009 (UTC)

I was experiancing this error earlier and was able to find out the source of my dilemma , durring my call for data from mysql, on the last like of the call there is "|data=your_variable=table_column,etc=etc..." my problem was I had removed the "data=" in call so it was "|your_variable=table_column,etc=etc..."

I hope this might help you -Brandon Collins

HI back with another error, trying to use the LDAP source:

[Thu Oct 15 22:56:59 2009] [error] [client 10.0.3.213] PHP Notice:  
Undefined offset:  1 in /wiki/extensions/ExternalData/ED_Utils.php on line 46, 
referer: http://wiki/index.php/External_Data_Testing

[Thu Oct 15 22:56:59 2009] [error] [client 10.0.3.213] PHP Warning:  
ldap_search() [<a href='function.ldap-search'>function.ldap-search</a>]: 
Search: No such object in /wiki/extensions/ExternalData/ED_Utils.php on line 
103, referer: http://wiki/index.php/External_Data_Testing

[Thu Oct 15 22:56:59 2009] [error] [client 10.0.3.213] PHP Warning:  
ldap_get_entries(): supplied argument is not a valid ldap result resource in 
/wiki/extensions/ExternalData/ED_Utils.php on line 104, referer: 
http://wiki/index.php/External_Data_Testing

thanks JS

Hi.... can you paste your #get_ldap_data call here? - Borofkin 06:03, 26 October 2009 (UTC)

{{#get_ldap_data:
 |domain=domain_def
 |filter=(sAMAccountName=username)
 |data=title=title
}}

{{#external_value:title}}

thanks -JS

CSV example table is garbled

I am trying to use the csv example below and the rendered tables is garbled as shown below. --RichardMcMahon 00:48, 29 November 2009 (UTC)

http://discoursedb.org/wiki/Fruits_table

Data was retrieved from the fruits data page, using the External Data extension.

All fruits (retrieved from the URL http://discoursedb.org/wiki/Special:GetData/Fruits_data, with links added to the fruit names as a demonstration):

{{#get_external_data:http://discoursedb.org/wiki/Special:GetData/Fruits_data|CSV with header|name=Name|color=Color|shape=Shape}}
{| class="wikitable" border="1"  cellpadding="4" cellspacing="0"
! Name
! Color
! Shape {{#for_external_table:<nowiki/>
{{!}}-
{{!}} {{{name}}}
{{!}} {{{color}}}
{{!}} {{{shape}}} }}
|}

Name	Color	Shape <a href="/research/vhs_wiki/index.php?title=Template:!&action=edit&redlink=1" class="new" title="Template:! (page does not exist)">Template:!</a>- <a href="/research/vhs_wiki/index.php?title=Template:!&action=edit&redlink=1" class="new" title="Template:! (page does not exist)">Template:!</a> Apple <a href="/research/vhs_wiki/index.php?title=Template:!&action=edit&redlink=1" class="new" title="Template:! (page does not exist)">Template:!</a> Red <a href="/research/vhs_wiki/index.php?title=Template:!&action=edit&redlink=1" class="new" title="Template:! (page does not exist)">Template:!</a> Round <a href="/research/vhs_wiki/index.php?title=Template:!&action=edit&redlink=1" class="new" title="Template:! (page does not exist)">Template:!</a>- <a href="/research/vhs_wiki/index.php?title=Template:!&action=edit&redlink=1" class="new" title="Template:! (page does not exist)">Template:!</a> Banana <a href="/research/vhs_wiki/index.php?title=Template:!&action=edit&redlink=1" class="new" title="Template:! (page does not exist)">Template:!</a> Yellow <a href="/research/vhs_wiki/index.php?title=Template:!&action=edit&redlink=1" class="new" title="Template:! (page does not exist)">Template:!</a> Oblong <a href="/research/vhs_wiki/index.php?title=Template:!&action=edit&redlink=1" class="new" title="Template:! (page does not exist)">Template:!</a>- <a href="/research/vhs_wiki/index.php?title=Template:!&action=edit&redlink=1" class="new" title="Template:! (page does not exist)">Template:!</a> Orange <a href="/research/vhs_wiki/index.php?title=Template:!&action=edit&redlink=1" class="new" title="Template:! (page does not exist)">Template:!</a> Orange <a href="/research/vhs_wiki/index.php?title=Template:!&action=edit&redlink=1" class="new" title="Template:! (page does not exist)">Template:!</a> Round <a href="/research/vhs_wiki/index.php?title=Template:!&action=edit&redlink=1" class="new" title="Template:! (page does not exist)">Template:!</a>- <a href="/research/vhs_wiki/index.php?title=Template:!&action=edit&redlink=1" class="new" title="Template:! (page does not exist)">Template:!</a> Pear <a href="/research/vhs_wiki/index.php?title=Template:!&action=edit&redlink=1" class="new" title="Template:! (page does not exist)">Template:!</a> Yellow <a href="/research/vhs_wiki/index.php?title=Template:!&action=edit&redlink=1" class="new" title="Template:! (page does not exist)">Template:!</a> Pear-shaped

Hi - the instructions should be clearer about this, but all you need to do is to create a page called "Template:!" that contains a single character = "|". Yaron Koren 14:19, 29 November 2009 (UTC)

Thanks, That fixed my problem. --RichardMcMahon 15:28, 29 November 2009 (UTC)

Special characters in data values

Hi Yaron

I am running (again :) ) into a unique situation.

I need to create links using values from an external data source. Problem is : these values contain both spaces and '[]' characters. This breaks the wiki syntax for creating links.

I tried using '{{urlencode' on values, but it seems that command is not interpreted inside '#for_external_table:' (all I get is the literal string '%7B%7B%7Bterm%7D%7D%7D' if 'term' is my variable from 'get_external_data'.

Any idea on how to resolve this ? or is that something missing in External data itself ?

- Laurent Alquier

I ended up adding a piece of code to ED_ParserFunctions.php to apply 'urlencode' to values of variables with a '.url' prefix in their name. So '|Term.url=term' will encode values of 'term' while '|Term=term' will not. I can send you the patch if you want it. - Laurent Alquier

Thanks for the idea! I implemented the same concept in the latest version, 0.9.2, but in a different way - variables are mapped normally, but then called using "{{{term.urlencode}}}". Yaron Koren 03:01, 11 January 2010 (UTC)

Cool. I prefer this approach a lot more ! Thanks for including it in your code. - Laurent

JSON

Hello! I'm trying to import some data from google translate api, which returns the results in JSON format. This is my code:

{{#get_external_data:http://ajax.googleapis.com/ajax/services/language/translate?v=1.0&q=hello&langpair=en%7Ces%7Cjson%7Cax=translatedText}}

{{#external_value:ax}}

I don't know if the URL is well encoded for External Data. I have tried all the posibilities I can think of, and nothing showed. Could you give me any hints? thank you very much

-- Jaime

Ooh, sorry about that - you've encountered a bug, for JSON pages that contain tags that aren't all lowercase. This will be fixed in the next version of External Data, but for now you can fix it in your local version by adding a line to ED_Utils.php - above line 258, which is:

                               if( array_key_exists( $key, $retrieved_values )

...you should add the following:

                               $key = strtolower( $key );

Yaron Koren 04:21, 7 December 2009 (UTC)

thank you very much Yaron!! it worked :) -- Jaime