Question.. architecting a large system. I’ve broken it down to microservices for the DB and rest API / gateway

I want there to be some some processes that run continuously not event driven via rest. Say analytics for example what is the best way todo that? Just another service running on on a server? And said service has its own API? That when the other rest APIs are called could then hop and call the new service?

Or say we had a PDF upload via rest should that service then do the parsing before uploading to DB .. or should the rest api that does the uploading then call another rest api to another service dedicated todo the parsing and uploading to the db?

I think the bigger way to explain the question is the encapsulation between DAL.. data access layer which I have existing.. but then there’s the BLL .. buisness logic layer which I don’t know if it should have its own APIs via own microservices running in the background.

Judging by your terminologies I'm going to assume you're using AWS. Also, take what I'm about to say not as gatekeeping but an effort to define that specific terms mean very specific things, names are important when describing ephemeral architectures.

First, it's not really MSA if you have a centralized database. True MSA utilizes dedicated databases for each microservice.

Second, REST is not event driven it's request driven.

When you say you want to run continuously, you could leverage ec2. I don't recommend that though. The whole purpose of going cloud native is not running 24/7. Instead leverage SQS, batch processing, or scheduled lambdas and S3.

In your example of uploading a PDF, use the S3 action hooks to trigger processing lambdas that enrich the data sent with the file as metadata when it was attached.

Keep in mind that API gateway has a filesize limit, last I knew it was around 5mb. If you want to attach files I recommend using S3 presigned URLs.

What you have described, to me at least, sounds like a very good candidate for an event driven microservices architecture. Take a look at CQRS patterns for microservices.

Using CQRS you separate the responsible layers of business, query, and command(write logic).

By using events you can trigger enrichment of the data with a business logic service that's completely separate from your data intake.

Currently the prototype is on a dev server using IIS, and sql server each MS has a separate database, bs logic to process results from different ms together for certain requests.

I agree with you regarding the terminology, I’m an embedded software guy c and c++ for the last 15 years.. before that I did stuff with web but nothing crazy. What I’m doing now is proof of concept working demo to pass onto the contractors eventually.
Have to have working concept to show value to then get funding from the upper management. Lol

Anyway my plan is to use Azure because we have a large commitment/ contract with “them” .. so I will be migrating to azure databases separate one for each ms .. and use azure functions run the api.

Do you think AWS would be better?

This is an intranet application no public facing side.

It’s very much an aggregation system of all our tools/systems for project management ALM/PLM style.

I only recommend AWS if you're well versed in the platform. It sounds like you're already locked into Microsoft as a platform though. Not a bad thing.

Just be aware of how Microsoft treats smaller customers.

Amazon has given me the same service on my personal account that they give my employers which manages a couple million dollars in accounts. There's some benefits they certainly get that I don't 😏 but the base customer service is comparable.

TLDR; stick with what you know, nobody will ask in 10 years why you used Azure over Amazon.

To me these sound like questions I would be asking after an exhausting day.

Get an evening of work-free thoughts, have a relaxing walk in a park and a good night sleep. Works for me every time - helps me to see things from a different angle, making all the answers obvious.

I don't think the question is "what's the right way". Imo it should be "what approach makes the most sense in your project's context: current and future development"

I would say to avoid servers altogether.
Use a service that allows you to run containers, like Fargate. Thus your api-facing processes just have to post jobs on a queue like SQS and your Fargate cluster can auto scale task containers that consume the queue. Each container just stops when the process ends.
You get more capacity elasticity this way, and no compromise on DevOps.

Do you have enough users / scalability problems to start thinking about microservices? That premise is not clear to me.

The analytics/reports suggests an cqrs approach I believe.

@gcavalcante8808 The application would be for internal use of the company. Global usage.

The application is a ALM/PLM, project management, systems aggregation platform.... its still in early concepts. mainly architecture and prototyping to eventually acquire approval for funding of the project.

Our current ALM/PLM solutions dont meet our needs for our workflow, and customer demands of things like ISO 26262 audits and ASPICE process evaluations... we have many tools we use and much of it is isolated and disconnected. which for trying to streamline processes we need to solution that ties it all together internally on our intranet.

@gcavalcante8808 further example, is currently we have a private github enterprise server, with global divisions using the platform, and the usage is "follow this standards document" and its very much wild wild west... one of the services will be is within the platform when a project is approved, a repo is created, permissions, branch management, would be controlled and automated thru the platform. to ensure all the repos meet the same workflow and approvals... again to meet audit requirements demanded by customers (automotive industry) (we are a global tier 1 supplier..... independent of my username lol)

@gcavalcante8808 another example is the project management (mech, elec, software, finance, manufacturing etc) is conducted via excel sheets, and open issues meetings... they are not the most tech savy and very old school... but the customer requirements and audits are causing a shift in the paradim and the software team is gaining more ground to find a solution that serves all not just our group ... while we still focus on our day to day projects....