AI Upscaling

[DarkRaven671]

As of version 2.6.0, AviSynth is no longer under development as it seems. There is now AviSynth+ as a successor. I'm a tech person, so I understand the challenge of working with either. There are no precompiled binaries for AviSynth+ available and AviSynth and AviSynth+ both use the same scripting language to control it. Compiling it yourself and write scripts is impossible for the average non tech person. The language looks fairly common for someone who has seen and worked with other languages though.

However, I'm not a video editing person. I don't really understand what AviSynth+ does and how it works. As far as I understand it doesn't edit videos by itself, but just serves individual frames to other software to further process. If so then it can't be used by itself to upscale videos, it just feeds the frames in a controlled manner into another piece of software that can do it. There are traditional (non-AI) upscalers available, so I guess it works together with one of these. I could be totally wrong here, as I said I'm not a video editor and don't know much about it.

The main focus as of now is upscaling supported by AI. The scepticism by aiwstq is somewhat justified, AI is often used as a sales argument to make your bullshit product look fancy and sell it to a unsuspecting customer. However, image enhancing by AI is a real thing. It's real technology in its early stages, but it's usable already. Topaz Video Enhance AI is the commercial product that most people, me included, are looking into right now. I own a license, so no junk included, but a free trial is available and that's also free of junk.

The software is clearly aimed at the average user with very little GUI elements. It's generally very low to the ground and mostly easy to understand for someone who's able to use a computer. It works by using several AI models (basically fancy trained algorithms) to analyze and enhance the image.

I don't know exactly how it differs from non-AI upscalers, as I said, not a video editor. But for sure it aims to be easier to use by an average person without the proper training. As it is often the case this could mean that the result is not as good as having a trained person doing it by whatever means they usually use.

One limitation is that the AI actually changes the video content. After processing it, it is technically no longer the same video, even though it obviously still shows the same content. Another limitation is the resolution of the input. While I haven't yet seen any official word on it by Topaz, my tests have shown that a very low resolution file, like 320x240 or 352x288 as you've mentioned, could be very difficult to upscale using this technology and the results aren't nearly as good as they are with slightly higher resolutions to start with. But on the other hand, this is all very much work in progress and maybe we see other algorithms in the future that are able to do it.

In the end this is all limited by raw computing power. The more you have available, the more amazing things can be done with it. To be a viable solution, especially for home users with limited resources, you have to make a cut somewhere. With modern day technology this is already very time consuming to do. That means as technology advances, so will likely the results that these algorithms can produce, in addition to the progress that is made on these algorithms themselves.

[DarkRaven671]

As it was said, this is still work in progress and there is constant development. Just got the news that Video Enhance AI 1.9.0 is out. Here are the changes for those that are interested:

Video Enhance AI now tackles deinterlacing

The latest update to Video Enhance AI (v1.9.0) introduces new AI models built for enhancing interlaced footage and eliminating those jarring, jagged lines produced by analog video recording. We call them our "Dione" models, and they can handle all types of interlaced footage while also denoising, sharpening and doubling frame rate of interlaced videos in one-pass:

Dione-DV is trained mainly to deal with high quality interlaced digital source such as DV, DVCPRO, DVCAM, etc.
Dione-TV is designed to handle analog TV/VHS/8mm sources.
Dione-TD is developed to be more robust for mixed frame type in DVDs.



We've also released Artemis v10, a new AI model that improves upon v9 and can produce output with better noise reduction, less residual color flicker, and fewer artifacts. Existing VEAI users can upgrade to v1.9.0 totally free.

I guess the de-interlacing part could be interesting for some folks. I'll check what the new Artemis v10 can achieve over the v9 and v8. Still waiting for more compute power than GTX 1070 can provide though to do this on a larger scale

[x4niUz]

Originally Posted by DarkRaven671

As it was said, this is still work in progress and there is constant development. Just got the news that Video Enhance AI 1.9.0 is out. Here are the changes for those that are interested:



I guess the de-interlacing part could be interesting for some folks. I'll check what the new Artemis v10 can achieve over the v9 and v8. Still waiting for more compute power than GTX 1070 can provide though to do this on a larger scale

Can't wait for my pc parts to arrive. Ryzen 5 3600 & 3060ti should be decent to upscale some.

[DarkRaven671]

That Ryzen CPU is nowhere near powerful enough to do any upscaling in a reasonable amount of time. You'll have to do it on the 3060 Ti and that will work to some extent, depending on the settings you want to use and how much material you want to process. It will still take quite a while though.

[x4niUz]

Originally Posted by DarkRaven671

That Ryzen CPU is nowhere near powerful enough to do any upscaling in a reasonable amount of time. You'll have to do it on the 3060 Ti and that will work to some extent, depending on the settings you want to use and how much material you want to process. It will still take quite a while though.

Not upscaling to 4k, just 1080p I think.

Should be at least better then what I have now :P Im still rocking the i5-2500k & GTX 970.

What CPU are u using then?

Time is fine, can maybe upscale on this one, once my other pc arrives. Just takes awhile which is fine

[DarkRaven671]

I just wanted to set expectations right in case you assumed you can do it on that Ryzen 5

My desktop uses a 24-core Xeon 8268 and that manages roughly 50% of the processing speed of the GTX 1070. I knew it would be too slow anyway, I just wanted to see how slow it actually would be.

Having a dedicated system to do this is probably a good idea if you want to do it on a bigger scale. That's what I'll probably do as well so that I'm not using the GPU in my desktop 100% all the time so that I can't play any games, for example. I've estimated that upscaling just the non-porn content would take between 1 and 2 years raw processing time with the GTX 1070 and I can't wait that long.

[DarkRaven671]

Topaz VEAI v2.0.0 was released this week and apart from some updated algorithms and general improvements it received an UI overhaul and looks a bit more modern now. While I'm writing this I'm removing some nasty compression artifacts from a video with Artemis v11 LQ and that seems to work really well. Still waiting for a new GPU though to do this properly...

In the meantime I've also looked into non-AI ways to improve video through reading about it in some video editing communites.

Some people are VERY vocal about this old-fashioned way being vastly superior and achieving the same or better results. These are mostly the "I've done it like this for 20 years" folks.

They say that this AI stuff is merely a sharpening tool in a fancy suit and they distrust anything that says AI on it (thanks to everyone who used the term to sell their bullshit and therefore burned it for everyone else) and Topaz in particular because of something they did (or rather didn't) 10 years ago when they failed to deliver something they had promised. Not sure what that was exactly, but it seems to be a personal thing for some people.

And finally, some are adamant about the difference between "improving" and "restoring" video. They clearly see themselves in the "restoring" business. Some refuse to work with compressed material and only work on raw source material. Which obviously is completely elitist behaviour because most of the time you don't have that luxury of having access to the raw material.

Some much more open-minded people confirmed though that the AI models indeed can produce very good results, comparable to or even better than other means.

[deak]

In the past I enhanced old clips by the "old-fashioned way" and I can tell you by no chance the old school will beat artifical intelligence: Not only is it sharpen, denoise, upscale, but if you have good quality material it can also produce more details than in the original. Now one could say this is manipulating the original content heavily but the results are very impressive!

[sslover]

Originally Posted by DarkRaven671

Topaz VEAI v2.0.0 was released this week and apart from some updated algorithms and general improvements it received an UI overhaul and looks a bit more modern now. While I'm writing this I'm removing some nasty compression artifacts from a video with Artemis v11 LQ and that seems to work really well. Still waiting for a new GPU though to do this properly...

.

Is the link for the Topaz VRAI? If I wanted to upscale a whole video, which one of these would I be using?

[DarkRaven671]

Originally Posted by sslover

Is the link for the Topaz VRAI? If I wanted to upscale a whole video, which one of these would I be using?

What do you mean by "which one of these"? Which one of what? Generally, in no way, shape, or form does the length of the video impact any of your choices that you need to make before starting to process the file. The length only affects the time that is needed to process it.