In T365154#9919019, @Yann wrote:

I wonder if the tool/system crashed on May 15th because of the number of videos submitted at the same time (more than a dozen feature films). Do you have any idea of the reasonable number of such videos the system can process at one time? Is there a queue/waiting list if too many videos are requested at once? I have more than 100 feature films to import...

If I'm understanding correctly, you're saying that if a client accesses a static asset from Commons as part of (in this case) a Wikipedia page view

In T322633#9895904, @Ebrahim wrote:

Just a drive-by comment, I was just wondering anything else can be adopted from a quick look at https://blog.pecar.me/sqlite-django-config and https://fractaledmind.github.io/2023/10/09/sqlite-on-rails-enhanced-sqlite3-adapter/

I think that is likely a good idea, but would like to defer to @Harej

Out of curiosity, are you intending to use this for something? Collabkit was never deployed, and the code is presumably quite bit rotted at this point.

The detection of third-party resources is still funky and I'd need to fine-tune the script but the initial version of the script generates the markdown table below.

With Wikimedia deployment being the end goal, I suppose putting it in core behind a feature flag would cause less friction than a whole new extension?

I would suggest making this an extension. I think this would be a very hard sell as a mw core feature.

As long as this isn't affecting availability and has no private information on it, I don't see what the benefit is of keeping this task secret.

Also, for reference, i tried to reproduce this, by installing 1.41.1 tarball, and doing wfLoadExtension( 'Auth_remoteuser' );$wgAuthRemoteuserUserName = "Everybody";, but i wasn't able to reproduce.

ES and FR crash.
EN, IT, RU work fine.

There are several other similar reports on Project:Support_desk, most of them saying it happens with no extensions just core mediawiki

In T365154#9873486, @Don-vip wrote:

Two of the files I submitted on 7th May popped on Commons in the last three days:

• .https://commons.wikimedia.org/wiki/File:KSC-04-S-00394_(ksc_120904_mode4).webm
• .https://commons.wikimedia.org/wiki/File:KSC-04-S-00397_(ksc_121304_rtf_stacking).webm

Does anyone else have seen one of his old submissions recently come live?

The weird thing is that the file appears as if it had been effectively imported on 7th May, which is not the case.

I honestly had forgotten this even existed.

Sounds good to me

That said, perhaps its time to archive this extension unless someone else wants to take it up. It was never deployed and development work has largely stopped afaik.

it isn't clear that CVE-2021-43519 has been addressed at all.

3 is basically the approach i took in the QuickInstantCommons extension (if you are loading images over network all these problems get so much worse). It was really hacky but the result was very effective. However im not sure it would be acceptable in core.

The biggest known issue at this point is the jobs aren't restartable, but they will be restarted if a deploy happens in the middle of the job (which will cause a failure). But i agree that is best served by other tickets.

Which search feature are you referring to in this task?

An alternate fix was merged (Still 7 chars, but use 7 even if ambiguous)

Well, I expect that bugs will a high priority are taken seriously.

This should be at least priority "High"

In T361956#9742426, @Tgr wrote:

In T361956#9724511, @Bawolff wrote:

With variables, the variable can be set as inline style

We should probably disallow that in wikitext ASAP (currently Sanitizer doesn't seem to filter it). It's a potential attack vector, and not just in TemplateStyles.

I guess i would check if the page table has any pages with a negative id. If not i would just delete the negative entries from page_props. Afterall it should be impossible to ever match the negative entries.

@hinnk Can you try again (in async) mode, and see what happens?

Oh, maybe this is a dupe of T340882 and there is an alternate patch there

Looks like we determine file name by doing git rev-parse --short=7 HEAD if 7 characters is not unique it does the shortest unique string.

File name is FlaggedRevs-REL1_41-447bfbc2a.tar.gz but https://www.mediawiki.org/wiki/Special:ExtensionDistributor?extdistname=FlaggedRevs&extdistversion=REL1_41 points to https://extdist.wmflabs.org/dist/extensions/FlaggedRevs-REL1_41-447bfbc.tar.gz not https://extdist.wmflabs.org/dist/extensions/FlaggedRevs-REL1_41-447bfbc2a.tar.gz

Hmm, i was going by caniuse saying that IE11 was the newest browser that didn't support it, which based on wikitech-l is soon to be grade X. I think thumbs not working in a grade X browser would be reasonable, but you raise a good point about the broader ecosystem.

I'm going to boldly re-open this for further discussion

@TheDJ if its just that file being broken, I think it makes sense just to leave it the way it is, so I'm going to reclose this bug (but please reopen if you disagree)

In T364644#9798733, @Ladsgroup wrote:

What would be the usecase for upload from other sites to non-commons. Assuming they are all fair use images (otherwise they should go to commons), maybe we shouldn't allow it? I know there are some discrepancies between copyright laws and some wikis allow some images that wouldn't be allowed in Commons but that's still quite a marginal usecase (unless I'm missing something super obvious). e.g. for T364288: Server-side upload request for hinnk why it's not being uploaded to commons?

I think IFD1 is for the embedded thumbnail (if present) and not the actual image. (ifd1 is irrelavent)

I suspect it wont even be that much more increase on job queue since upload wizard is disabled on other wikis. This would entirely be about third party tools which is probably just a few people.

In T355725#9788215, @Harej wrote:

In T355725#9788183, @Harej wrote:

• Have a maintenance script that retrieves, from the Wikidata Query Service, a list of Wikipedia articles connected to items that are associate with MDWiki via P11143.

My own counterargument to this is that the Wikidata query to retrieve this information is very expensive:

SELECT ?item ?value ?article
WHERE {
  ?item wdt:P11143 ?value.
  ?article schema:about ?item;
           schema:isPartOf [ wikibase:wikiGroup "wikipedia" ].
}

This times out on the regular Wikidata Query Service, and even on my own Wikidata Query Service it takes several minutes. Sure, you don't need to run the query all that often, but if an error or vandalism got in, it wouldn't be removed until the next update. Even if you forced an update, it could still take several minutes for the query to return. Not ideal.

I ended up just reading the exiftool source code.

It would be useful to link to (or upload to this task) some example files with XMP [As a reminder, many devs do not actually have the ability to view deleted files on commons]. So far, I've noticed that image magick seems to diverge from the spec when creating webp files, so it would be good to have some examples from a variety of tools just to see what they do. (I guess i don't need this anymore, after looking through exiftool source code)

Why ICC? I don't think we would normally show that type of data on a file description page?

I have no idea if anyone is still around who knows the history of why this is the way it is. @Sannita does the current upload wizard team have any thoughts or objections to changing this?

Ah, that would be the problem. Async uploads are disabled on english Wikipedia - https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/InitialiseSettings.php#L729-L735

Hmm, both of those errors on the server side indicate no async. So either something is wrong with the script or maybe enwiki somehow has async disabled

Also broke the techblog link on mediawiki.org

@hinnk How are you uploading these files? (Are you using UploadWizard, a specific user script, a bot program, something else?) I suspect this may be an issue with the tool being used to do uploads.

I think there are two separate tickets here:

So recently i've been trying to collect information on cases where large uploads fail in order to try and fix the underlying issues. Did you attempt to upload this to commons in the normal way and it didn't work? If so, please let me know the details.

I think if we did deliver the wrong thumbsize, it only makes sense to deliver one larger then the requested size. Browsers will scale it client side, and its always better to downscale then upscale. The performance degredation is probably be not that bad - the larger thumbs would only be shown to first person to view the page. A small slowdown to image viewing for a single user seems not that bad, and better then never loading the image at all.

Possibly the value "auto" should also be treated as "100%"

@Aklapper did you mean to direct this to @Halfak ?

There are no interesting changes between 643fb6c (what mdwiki uses) and current master of TextExtracts. So either its a configuration difference or a change in a different component

Are you reporting that this used to work but no longer does (In which case when was the last time it worked?), or has it always been this way and you are suggesting that the module should be changed to work as you describe?

the patch at https://pastebin.com/9wVWHEpa lgtm

Maybe should just be limited just to executable files

SVG is like HTML - there are dangerous parts but its not that big a deal. We allow <div> in wikitext but not <script>, we could do the same thing with svg potentially.

But if we don't find a replacement my assessment is that we will eventually end up in the same situation with timelines as we currently are with graphs.

Maybe, but i would argue a new task would be better, when and if that happens.

my understanding is that this worked after retrying.

Seems like this is fixed! We just had File:OBR_Hafignnover_6-2024.webm (4.88 GB) successfully uploaded (req-id: c1a5aab7-6e6b-4ef5-b20d-f9ddb095577f ). The job took 5 minutes 20 seconds to complete, so went beyond the previous 202 second limit.

Tagging for 1.39 release because there was a request on matrix to backport to REL1_39 (As in make mw core rel1_39 pull in a fixed version of library). I'll leave it to release managers if they want to do it.

per T358308 - one of the major causes of upload failure for medium-large files, is that during a deploy the job runners get killed. Making it retryable would fix that issue.

Can you try uploading again. The issue might be fixed now

Yes, this is due to T358308. Until that is fixed, you can upload large (>3GB) files via https://commons.wikimedia.org/wiki/Help:Server_side_upload