Did development start yet?

>>11014
In a way it's been started for a while.
Software languages(Golang only), libraries (Gin, Pongo2, MySQL) and development process established(Test heavy/driven development).
Use case list needs to be written so I can't start the code yet.
Then I have to make some architecture related decisions which will decide where all the code files go.

So still in pre-alpha. Multiple "prototypes" have been written. I might want to do something else to get a few design patterns understood, but probably not necessary and can just wing it.

Is it fun to write an imageboard engine?

I wouldn't call it super fun, but I enjoy rigging things up on the backend. I loathe how webdev is these days so I enjoy finding clever solutions to problems on that end too where most would just use JavaScript and 700 libraries instead of clever HTML and CSS hacks.

>>11015
Why not use more of the standard library? Go houses some powerful features right out of the box, including a HTTP server and a HTML templating engine. It's two less dependencies you have to worry about.
I know net/http and html/template can be cumbersome at points and getting the initial handler/templating layout can be a little hard at first but it works damn well once you get it to work.
Then again, I've only ever used that and the Fiber framework.

How are you designing the engine? Doing database stuff first and testing where possible, then tossing everything else on top?

>>11021
Gin looks to be similar to Fiber.
Gin and Pongo2 has lots of usage in major websites. Only issue I've come across is that fuzz testing with Gin is very inefficient. Just can't do it without re initializing the server every round. So basically I have to abandon one of the modern aspects of Golang's standard testing library. Why not use the standard net/http, just because I've tried doing that with the Kissu's server which generates the JSON pages(amoung other things) and it ended up being a mess. Not going to reinvent the wheel.
As for Fiber. I don't like the Express engine it's based on(it's what serves these pages to you and generates the JSON which client side renders these pages). Something about the handlers didn't stick with me.

html/template is fine. It's alright, but I might as well go big. Also I can't find any information about using custom functions inside of templates with html/template while Pongo2 allows this. Pongo2 and Vichan's Twig are similar in that they use functions inside of templates regularly, but yet it will still require a bit of effort to translate between the two. There is a Golang Twig analog called Stick and I used that in the feedback engine, but it's lacking in support(still not a finished product) and studies. Some other alternatives such as Amber are a complete departure from Twig's syntax.
https://medium.com/@kataras/whats-the-fastest-template-engine-in-go-fdf0cb95899b
Blocks is not very active but an alternative I guess.

I said, pretty clearly that I'm not going to be amateurishly hacking everything on top. Pretty rigid adherence to the practice of TDD. Maybe the routing will be done with manual testing.
Database will still be MySQL, but I figure it's time to get rid of posts_b, posts_jp(and it's files json entry attatched to each post) and start a more by-the-book relational database instead of the existing no-sql and relational hybrid.

The order by which I'll be doing things I'm not sure about. I need to build up what will likely be a 100+ item use-case list covering things from Vichan and my support servers which will be integrated into this future imageboard engine. Then I will write up some architecture diagrams, somewhat resembling UML but not really, just a way for me to collect my thoughts.
When all those are done, I begin writting to fill in the blanks until I have a Beta product which I'll put onto the site to do active testing.
This is the system that I did to create the UI, but this time it will be after having experienced issues with my development process on the UI and a more professional take on understanding what I'm getting into before I begin writing. More planning, less hacking.

>>11022
Fiber is good for getting things up and going fast, but I agree with your criticisms. I don't hate it, but I wouldn't use it for anything serious that I would start today; I have one project in it which works, but I occasionally found myself beating my head against the wall because of Fiber.
net/http is barebones but you can build your functionality on it at least and only include what you need. My most major project in it uses regular expressions to determine which route to take, but it also has only three routes anyway.

>using custom functions inside of templates with html/template
You can define functions in Go and use them quite easily; allow your template to use them by tossing them into (*Template).Funcs, functions that have a return type of template.HTML will not be HTML escaped but all others will.
Inside of templates is a bit weirder and as such I don't really bother with it, but essentially {{define "template name"}}do stuff with {{.}}{{end}} and call with {{template "template name"}} or if you want to set dot, {{template "template name" <dot value>}}.
However, by the time I'm doing that, I'm hitting the absolute limits of html/template anyway so I just write the same thing in Go, where it would be faster regardless of the templating engine, except quicktemplate which is a different beast altogether.

>performance
Don't worry about super fast performance until you're getting those numbers of requests per second, in which case you're likely getting hit with an attack anyway.
I would assume that most would not notice the few hundred microseconds on a slightly faster engine either.

>I figure it's time to get rid of posts_*
You should keep them to reduce query complexity, but I would move the files JSON into the database. Unless it already is, in which case I now wonder why you would toss JSON into the database in the first place, but I digress.
My imageboard projects never made it outside of my machine (save one but it's a textboard), but I had no qualms with a {posts,files,replies}_board table; don't take this from me however, as I am by no means an expert on anything I do, nor is everything I do right and best.

>>11023
>(*Template).Funcs
ah, i see now.

No, I'll worry about performance and making it work quickly for as long as the site expands. This isn't slight either, it's double as much throughput. The only argument you can make is that Golang with make a PHP-esque change and fuck up every dependency that relies on it.

Files is stored as json inside of the posts_* entries.
Increased query complexity is important because doing lookups to build /all/ is monstrously expensive because I have to write 9 different union statements for each board.

This Funcs system is pretty ugly.
https://www.calhoun.io/intro-to-templates-p3-functions/

Compare it to this pongo2 example.
https://github.com/ECHibiki/Kissu-Feedback-and-Forms/blob/main/src/server/templater/pagegen.go
I prefer this system more. Assigning functions to the template engine when you init is better than doing it at runtime, yea

>>11024
>Increased query complexity is important because doing lookups to build /all/ is monstrously expensive because I have to write 9 different union statements for each board.
If you're set on getting rid of the several boards, I recommend you check out the source to Picochan which does just this. I can upload if you so wish. It's a Lua CGI SQLite3 onion imageboard that I do not frequent yet have the source of, which I have combed through enough to tell you that this one table scheme what it does.

I figure it would be monstrously expensive for the overboard regardless of what you do, but I'll dump in my suggestion regardless: I wonder what the performance would be if you were to do it automatically.
Create a table housing a thread's ID, its board, and its last bump. A trigger automatically creates, updates, and deletes from this table as needed.
Pull from this table and sort by the last bump date and gather only what you need with pagination.

I may try this myself with some dummy data soon for my own projects.
I'll sleep before I start sounding more like a fool, thank you for your time.

>>11026
I don't know why you'd suggest all this stuff when a site like pornhub literally runs on a MySQL fork(wish I remembered which one) and does fine.

Another point that has come up where posts_b, posts_jp is causing issues...
When a post is moved between boards, or just moved in general, it requires a recreation of the existing thread and then a deletion of the old one.

With a new system it only requires duplication(in the case of cloned moves) or a change in a datafield with where it belongs.

There is very little reason to have there be board specific database tables other than the thought that it will make sites go a little bit faster on posts(of which image manipulation is the most computationally expensive task anyways...)

I'm going through all of Vichan and my own stuff's functions and writing out use-case lists which I'll use during my architecture formulation stage.

This use-case list will be open source, basically encapsulating all of kissu-vi's functionality and things I plan to add to Tsukuyomi
https://github.com/ECHibiki/Tsukuyomi-UseCase-List

I'm writting into my draft every time there is a delete action that they will be stored in an inaccessible recycling bin for a few hours. This will allow for moderator controlled rollbacks on some of our actions which are normally irreversible.

These would be true deletes(user deletes), page renumbering deletes, archive deletes, cyclical deletes.

A proposal I have to add in a long term text archive of kissu would be connected to the concept where some of these recycling bin items that are cleared from recycling would be accessible on another database. This would be a lightweight version of the archives we know well on 4chan, but on Kissu.
Ghostposting you might ask? We'll think about it.

Finally starting on graphing out the program.

One of the more advanced things which I don't think any other imageboard software does is security checks on startup.
Tsukuyomi will:
- Check if files and folders are at safe permission levels (Nothing should be root 777)
- Check that settings files are not accidentally being read by NGINX file rules

>>11121
What program is that? Looks pretty useful

>>11122
GitMind. https://gitmind.com/
Very general purpose program.

When I was initially drafting the UI I used some terrible thing which we used in university. This is leagues better

I don't think I actually have the graphs on local storage though. Haven't checked. Seems like cloud data.

https://gitmind.com/app/docs/flwp22tg

I'm considering potential ways to use this trick
>>>/qa/101227

one optimization coming to mind is that this can be used on thumbnails to reduce storage in the archive concept I want to add.
Another is that post times could be increased greatly by evaluating a blockhash of a submitted image... such that there's no need for image processing...

Very interesting trick

Almost at the first draft of the diagram stage.
Have to describe what each of these SQL tables will be doing and their FK associations

Effectively our first draft.

Won't make any sense to anyone but me yet.
My next task is to go correct issues, expand missing details, arrange the elements.

Following that will be to write out the pseudo code... text based summary of the operations... and following a similar process as the diagram stage.

Following this... the conversion of pseudo code into software(writing out code from text descriptions) is getting to a point where AIs can assist programmers to create buggy, insecure and unmaintainable code.

A copy of my draft is: https://gitmind.com/app/docs/f2r64yan
Will be my documentation only github shortly: https://github.com/ECHibiki/Tsukuyomi-UseCase-List

hm, for some reason it didn't add labels to the SVG export.

What a modular abomination

Progress on the rewiring and finalization of the blueprint.
After a break will do the first line of packages which will complete 60% of it.

A bit neater.

Think I will have to divide up one of the sections into a more detailed view. Or maybe not. I want to get into pseudo coding by the New Year so I might fill in some blanks as I go.

There's more that I could do with this, but it would get to the point where it's getting a bit obsessive.

https://gitmind.com/app/docs/f6pvzikf

I'll just leave it at this for now

Christmas is over. back to work tommorow

It would be fun if the characters you had to pick for the flood filter captcha were updated with some new additions. How about Kuon and Lala?

yeah. maybe it's time to update those. Still have to move some software around. Can fit that into then

Going to end up having to redraft these plans to add in a socket server.

bleh. thinking of the site's software and on the differences between golang and rustlang, there's a case for the main services of the site(the imageboard engine) to be running the machine level rustlang. Removing the Golang's GC (as apposed to the hastle of dealing with it) will end up netting better results for the site than a faster development time through golang, since there's a risk that continued development on a golang server will hit a wall in golang's memory management limitations.

However, the standalone programs (things such as the banners program, captcha program, theatre program or feedback program), these are small programs that have little relation to the imageboard activity and benefit more from the simplicity of scripting languages and GC.

So, this is to say that the thoughts of adding in a socket server and switching some of the UIs operations to sockets has made me consider potential issues with the imageboard engine as it adopts to more complicated requirements. I want to premptively respond to any crazy thoughts I have.
So far, there are little issues with the vichan server aside from it being outdated and hard to customize. So I'm comfortable with the added time it takes to develop in this language.

>>11577
Set of graphs for site architecture changes

Instead of rewritting the IRC bot for some practice, will temporarily split off Vichan's image generation into a seperate rustlang binary which will reduce the amount of PHP bash/shell_exec calls which are not very performant.

- Rustlang thumbnail component implementing the API for FFMPEG, Imagemagik.
- Will need to find a solution to exifdata
- Some imageban stuff might be kept.
- should result in better post times on images.
- Component that will be easily portable to rewrite

Basic idea for the rewrite to maintain visible progress while also progressing towards a major goal will be splitting off some more sections to make the task easier.
Basically already at that point but I need a bit of work on Rustlang stuff to get to that point.
Most of the site's operations are split into miniservices, but the post and mod operations remain.
Some new ideas with the sockets and IM posting will require it to be rewritten rather than kept on life support.
But still, the posting operations are the most major aspects of the site.
With the offloading of image posting onto a new service the task will be easier.
My hope is that by the time we've split off the image aspects then moving a bunch of Golang and Rustlang code into a Rustlang monolith application should be straightforward.
Issues about scalability will be met by a multiple monolith approach where the multiple servers are communicating with one-another through websockets. This way a client with a socket connection to a /qa/ server would be able to still get information from a /jp/ server.
In program memory management and analysis to provide limitations in process consumption.

The ideas of this approach I'm taking can be read about here
https://blog.twitch.tv/en/2022/03/30/breaking-the-monolith-at-twitch/
But while they wanted to split it up, opting to glue together. Essentially creating a standalone program that requires a minimalist operating system and nothing other than the ability to run binaries

>To those who wonder what might be wrong with the aforementioned approach, here's a hint: NO, that's not how software is written. First, you sit down and study computer science, algorithms (Knuth before all), programming ideas, principles, and paradigms. Then you analyze your task and explore and evaluate the tools needed to do it, google around, ask for tips and pointers. Then you pick your tools (PHP+SQL in our case) and study them thoroughly by reading technical literature available in rich selection. Then you make up a complete plan of what you are going to do, and how exactly, usually employing the "divide and conquer" principle by breaking up the task into logically orthogonal interacting units. And only after all of that you actually start writing code unit by unit. And finish with testing (+ regress-testing for updates), debugging, profiling, and optimizing. At least that's how I've been taught... But apparently not this shitshow crew behind Tinyboard and vichan.

Such a quote is very nice reassurance since it's my approach. But it's only common sense really for anyone who has formal education in computer science

His statements that using Twig in PHP is retarded is also true.

The same sentiments came to mind when thinking about how to remake vichans templating engine. That it's probably to little benefit to even bother and all languages seem to have good enough built in templating ability.
Tenplating engines just abstract out some of the architecture complexity

we're going to heavily use .so files to create short term replacements to vichan's php components. This provides an easier solution to the issue of new server software allowing me to get a better idea of what I'm getting into by doing this project.
This and providing a speed boost to existing operations and giving us a 'security by obscurity' situation, walking us away from the well known and understood vichan.
Still want to create a large kissu imageboard engine, but it's time investment will be too large to ignore certain issues without smaller fixes... plus I don't want to go into a situation where I'm always thinking that it needs a few more years to be perfect

Doesn't look like you can do serious extensions of Vichan with PHP-FPM using .SO files.
You run into issues pretty quickly when trying to use it to connect with MySQL. Segfaults. Maybe if I wrote a mysql library from scratch I could find out if it's a library issue or an FPM issue.
I'm doing this for educational purposes in building Rust servers so I might as well find out.

So the way things seem to work is that the gradual transition approach is not a favorable strategy.
Can probably replace things like the templater individually. Also mod.php but since it touches the database you have to do the entire file all at once which is a larger commitment than I was hoping for.

Likely what I'll do before trying to create the monolithic imageboard engine is continue removing parts of our existing software with components and ideas that will be used later:
- Track down the specific .SO-MySQL issue
- Rewrite the templater using a from scratch Rust implementation
- Upgrade/Remake search.php with a Rust implementation and some AI tools
- Implement the IM service
- Rewrite some other segments of code(NodeJS) parts for some extra knowledge
Other than Generative AI goals which could pop up, that's probably all that needs to be cleared before I feel like I can do an S tier job with writing a server.

This dude's been hitting my feed.
And my Rust+PHP things didn't pan out. Also doing a ton of golang stuff now and I'm seeing that what I liked about Rust might not actually exist in reality.
The things Meguca thought when deiciding between go or rust were kind of wrong and focused more on high performant websockets. But you don't actually need websockets for most things, can just use HTTP/3 unidirectional sockets.

If anything we'd want client side sockets for IM/IRC stuff and I'm kinda actually satisfied with my hacked together Rustlang Sageru-Kissu relay. Perhaps can touch it up a bit.
To be honest, the front-end of kissu is basically complete. I can't think of anything more to do with it. Even if I did put unidirectional sockets in it would just be to reduce bandwidth usage for mobile devices.
Problems/Grievances with Kissu are hardly the fault of a lack of features unless people can prove otherwise.

It really is amazing how I can do bulk post moves and the post reference (quote link) is rebuilt, and it happens in seconds. I did the happenings thread -> reddit move of about 20 posts at once, including images, and it completed in seconds. Images were rebuilt and labeled with the m- prefix and it's just so cool. It's not something regular users can ever see, so I just wanted to point it out

>>11800
As a user, I also think it's pretty cool. Short comment, but yeah.

yeah its real cool i remember that one time a big move like that broke all the links its much appreciated
i THINK >>>/qa/109652 was pointing to the posts before the move and wasnt updated...

>>11802
yeah, it doesn't cascade like that.
I don't really believe that imageboards or anything involving user content should be put into SQL databases. Just makes this entire thing a pain to deal with.

SQL for ordered data presentation and analytics.
Document QL for variable data structures

>>11803
What type of database does Kissu use?

>>11804
MySQL
https://github.com/vichan-devel/vichan/blob/master/inc/config.php#L90

My ideal here is the following, but starting with new plans will take a bit of time.

Mod UI = dynamic binding QT C++
- Separate the moderator activity for better security practices
- I want to be able to make GUI again and potentially pick up my OpenGL knowledge(therefore there is no alternative to C++)
- Good stepping point for other desktop application ideas
API Server = HTTP2 Golang-Gin server interfacing with DB and Cache
- A simple REST API that issues information for clients
- Make use of HTTP2 websockets in place of certain types of polling
- Better progress feedback on uploads
- Instead of JSON files use Redis cache
- Vichan is built on JSON files already so switch to MongoDB
- Maintain legacy HTML pages
WASM UI = Redo with Solid instead of React
- Stay hip and with the times
- Opportunity to clean up past spaghetti and add new features

This is probably the best plan that gets me skills in software engineering and removes the 5 years of outdatedness that kissu's accumalated.
And I can do them individually at a slower pace while keeping things running on the site.
Build the Go API handling for mod routes, then have a C++ application interface with it, then when that's done shift the site over to SolidJS and then change API server to start using MongoDB or whatever else I decide to be good.

I'm thinking that for this goal we're going to be looking into porting the Vichan SQL over to another database. I'm thinking Postgres but open to looking into other suggestions people bring up if they want

>>11834
Adjustments to this planning

ModUI: Tauri based GUI
API Server: HTTP2(3 ideally) Golang-Gin + Gorilla server
UI: Existing kissu-UI with a plan for a Tauri UI

probably MariaDB makes more sense, thinking of it a bit more and using Opus to give me some comparisions...