You can use the singleTop launch mode to get the behavior you want.

From https://developer.android.com/guide/components/activities/tasks-and-back-stack:

If an instance of the activity already exists at the top of the current task, the system routes the intent to that instance through a call to its onNewIntent() method, rather than creating a new instance of the activity. The activity can be instantiated multiple times, each instance can belong to different tasks, and one task can have multiple instances (but only if the activity at the top of the back stack is not an existing instance of the activity).

For example, suppose a task’s back stack consists of root activity A with activities B, C, and D on top (the stack is A-B-C-D; D is on top). An intent arrives for an activity of type D. If D has the default "standard" launch mode, a new instance of the class is launched and the stack becomes A-B-C-D-D. However, if D’s launch mode is "singleTop", the existing instance of D receives the intent through onNewIntent(), because it’s at the top of the stack—the stack remains A-B-C-D. However, if an intent arrives for an activity of type B, then a new instance of B is added to the stack, even if its launch mode is "singleTop".

I changed my showNotification to this

public void showNotification(String Title, String info){
    int mNotificationId = 1;

    final Intent notificationIntent = new Intent(MainActivity.this, MainActivity.class);
    notificationIntent.setAction(Intent.ACTION_MAIN);
    notificationIntent.addCategory(Intent.CATEGORY_LAUNCHER);
    notificationIntent.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK);

    PendingIntent pendingIntent = PendingIntent.getActivity(MainActivity.this, 0, notificationIntent, 0);

    NotificationCompat.Builder builder =
            new NotificationCompat.Builder(MainActivity.this, "ALERTS")
            .setSmallIcon(R.drawable.notification_icon)
            .setContentTitle(Title)
            .setContentText(info)
            // Set the intent that will fire when the user taps the notification
            .setAutoCancel(true)
            .setContentIntent(pendingIntent);



     mNotificationManager.notify(mNotificationId, builder.build());
   
}

And also added singleTop in the Manifest like @Maurice Lam suggested.
Now it seems to be working correctly.

I am building an Android app which should just open the app once its notifications are clicked.

    private void createNotificationChannel() {
        // Create the NotificationChannel, but only on API 26+ because
        // the NotificationChannel class is new and not in the support library
        // Gets an instance of the NotificationManager service
        mNotificationManager = NotificationManagerCompat.from(this);
        if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
            CharSequence name = getString(R.string.channel_name);
            String description = getString(R.string.channel_description);
            int importance = NotificationManager.IMPORTANCE_DEFAULT;
            NotificationChannel channel = new NotificationChannel("ALERTS", name, importance);
            channel.setDescription(description);
            // Register the channel with the system; you can't change the importance
            // or other notification behaviors after this
            mNotificationManager.createNotificationChannel(channel);
        }
    }

    public void showNotification(String Title, String info, int id){   
        // Create an Intent for the activity you want to start
        Intent resultIntent = new Intent(this, MainActivity.class);
        // Create the TaskStackBuilder and add the intent, which inflates the back stack
        TaskStackBuilder stackBuilder = TaskStackBuilder.create(this);
        stackBuilder.addNextIntentWithParentStack(resultIntent);
        // Get the PendingIntent containing the entire back stack
        PendingIntent resultPendingIntent =
                stackBuilder.getPendingIntent(0,
                        PendingIntent.FLAG_UPDATE_CURRENT);


        NotificationCompat.Builder builder =
                new NotificationCompat.Builder(MainActivity.this, "ALERTS")
                .setSmallIcon(R.drawable.notification_icon)
                .setContentTitle(Title)
                .setContentText(info)
                // Set the intent that will fire when the user taps the notification
                .setContentIntent(resultPendingIntent);


        mNotificationManager.notify(mNotificationId, builder.build());
    }

I followed this guide: https://developer.android.com/training/notify-user/navigation

The notifications display correctly and once I click them it takes me to MainActivity, but it restarts MainActivity. Why does this happen and how can I make it to just open it as if opened from onResume?

[EDIT] added my XML structure

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns_android="http://schemas.android.com/apk/res/android"
    package="com.company.example">
    
    <application
        android_allowBackup="true"
        android_icon="@mipmap/ic_launcher"
        android_label="@string/app_name"
        android_roundIcon="@mipmap/ic_launcher_round"
        android_supportsRtl="true"
        android_screenOrientation="sensorPortrait"
        android_theme="@style/AppTheme">
        <meta-data
            android_name="com.google.android.gms.ads.APPLICATION_ID"
            android_value="..."/>
        <activity
            android_name=".MainActivity"
            android_label="@string/app_name"
            android_windowSoftInputMode="adjustPan"
            android_launchMode="singleTop"
            android_exported="true">
            <intent-filter>
                <action android_name="android.intent.action.MAIN" />
                <category android_name="android.intent.category.LAUNCHER" />
            </intent-filter>
        </activity>
        <activity
            android_name=".Settings"
            android_label="Settings"
            android_exported="false"
            android_parentActivityName=".MainActivity"
            android_theme="@style/AppTheme.NoActionBar">
        </activity>
        <service
            android_name="com.company.example.DeviceService"
            android_permission="android.permission.BIND_MIDI_DEVICE_SERVICE"
            android_exported="false">
            <intent-filter>
                <action android_name="android.media.midi.MidiDeviceService" />
            </intent-filter>

            <meta-data
                android_name="android.media.midi.MidiDeviceService"
                android_resource="@xml/device_info" />
        </service>

    </application>

</manifest>