🎉 New! Web Push Notifications for Chatkit. Learn more in our latest blog post.
Hide
Products
chatkit_full-logo

Extensible API for in-app chat

channels_full-logo

Build scalable realtime features

beams_full-logo

Programmatic push notifications

Developers

Docs

Read the docs to learn how to use our products

Tutorials

Explore our tutorials to build apps with Pusher products

Support

Reach out to our support team for help and advice

Sign in
Sign up

Building a media conversion server for a React Native chat app

  • Wern Ancheta

August 21st, 2019
You will need to have Node 11.2+, Yarn 1.13+, React Native CLI 2+ and React Native 0.59+ installed on your machine.

Across the various devices and platforms your app runs on, there are different file types created for the same type of media. When connecting users across platforms, it can be incredibly hard to build sharing features due to the varying compatibilities of different file formats.

In mobile, this problem is especially problematic as you don’t have control over which software is running on your users’ device and which file types are natively supported.

In this tutorial, we’ll be looking at using an intermediate media conversion server to alleviate this issue. We’ll use a chat app with sharing as an example, but the techniques you learn here will be useful to any other kind of app where sharing various file formats across platforms is required.

Prerequisites

Basic knowledge of Node.js and React Native is required to follow this tutorial.

Chatkit is used for providing the chat functionality.

Ngrok is used for exposing the server to the internet.

The following versions is used for building the app. Be sure to switch to those versions if you encounter any issues in running the app:

  • Node 11.2.0
  • Yarn 1.13.0
  • React Native CLI 2.0.1
  • React Native 0.59.9

This tutorial assumes that server will primarily be a Linux server. But it will also include instructions on how to set up the required software on Windows and Mac operating systems whenever possible.

App overview

The app that we will be working with is a chat app that allows the users to attach different kinds of files to a chat message. Specifically, it’s going to handle the following conversions. Only the following were tested in this tutorial, but you shouldn’t have any problems converting any other file type as long as the underlying software supports it:

Media type Convert from Convert to
Documents epub, docx, odt pdf
Audio mp4, ogg mp3
Video flv, mov, avi, webm mp4
Image webp png (jpeg and gif also supported)

To ensure that the app can render the different file types, we will create a media conversion server with Node.js. This server will be responsible for converting the files into a format that the app can understand. It will also serve as the host for the files. So it will be hosted on your own server instead of having it on Chatkit’s servers.

Here’s what the app will look like:

You can find the source code on this GitHub repo.

Setting up the server

In this section, we will be setting up the media conversion server. As mentioned earlier, we will be working with image, document, audio, and video files. So we have to install all the software required for converting between different file formats.

First, install ffmpeg, the cross-platform solution for recording, streaming, and converting audio and video files:

    sudo add-apt-repository ppa:jonathonf/ffmpeg-4
    sudo apt install ffmpeg
    ffmpeg -version

Next, download and install the latest stable release of Pandoc. Pandoc allows you to convert almost any type of document to another. In this case, we only use it for converting document files to PDF. At the time of writing this tutorial, the current version is at 2.7.3. These are the commands for installing it:

    wget https://github.com/jgm/pandoc/releases/download/2.7.3/pandoc-2.7.3-1-amd64.deb
    sudo dpkg -i pandoc-2.7.3-1-amd64.deb

Next, we need to install wkhtmltopdf. Pandoc depends on it to convert HTML to PDF. So Pandoc will actually only convert the files to HTML, then it uses wkhtmltopdf as its engine to convert the HTML to a PDF:

    sudo apt-get install xvfb libfontconfig wkhtmltopdf

For converting images, we’ll use ImageMagick:

    sudo apt-get install imagemagick

For Mac, you can install the required software using the following commands:

    brew install ffmpeg
    brew install pandoc
    brew cask install wkhtmltopdf
    brew install imagemagick

For Windows, you can find instructions on how to install ffmpeg and Pandoc on these links:

wkhtmltopdf has its own installer which you can download here. Don’t forget to include it to your environment variables so the system recognizes it.

Bootstrapping the app

So that we can focus on building the relevant features of the app, I’ve created a starter project which already contains the basic chat app code and server code. Go ahead and clone it and install the dependencies:

    git clone https://github.com/anchetaWern/RNChatkitMediaServer
    cd RNChatkitMediaServer
    yarn
    react-native eject
    react-native link @react-native-community/async-storage
    react-native link react-native-config
    react-native link react-native-document-picker
    react-native link react-native-gesture-handler
    react-native link react-native-permissions
    react-native link react-native-vector-icons
    react-native link react-native-video
    react-native link react-native-pdf

React Native Config has extra steps you need to follow to configure it properly:

React Native Audio Toolkit can’t be configured via react-native link so you have to follow these steps from their official docs:

You also need to install the server dependencies:

    cd server
    yarn

Lastly, update the .env and server/.env file with your Chatkit credentials:

    CHATKIT_INSTANCE_LOCATOR_ID="YOUR CHATKIT INSTANCE LOCATOR ID"
    CHATKIT_SECRET_KEY="YOUR CHATKIT SECRET KEY"
    CHATKIT_TOKEN_PROVIDER_ENDPOINT="YOUR TOKEN PROVIDER ENDPOINT (only for chat app)"

Building the server

Now we’re ready to add the functionality for handling file conversion to the server. As mentioned earlier, the server already has the code for handling the chat functionality so all we have to do is add the code for handling file conversions.

Start by importing the modules we’ll need:

    // server/index.js
    const filetype = require('file-type'); // for determining the file type of the uploaded files
    const fs = require('fs'); // for reading/writing to the filesystem
    const multer = require('multer'); // for handling file uploads
    const getStream = require('get-stream'); // for getting the stream of files
    const randomstring = require('randomstring'); // for generating random filenames
    const ffmpeg = require('fluent-ffmpeg'); // for converting audio and video files
    const nodePandoc = require('node-pandoc'); // for converting between different document types
    const gm = require('gm').subClass({ imageMagick: true }); // for processing images with ImageMagick
    const mimetypes = require('mime-types'); // for getting the extension based on the mime type

Next, initialize the array which contains the file types that are accepted by the server:

    const valid_filetypes = [
      'mkv', 'mp4', 'avi', 'flv', 'mov', 'webm', 'wmv', // video files
      'wav', 'flacc', 'mp3', 'ogg', 'm4v', // audio files
      'jpeg', 'jpg', 'png', 'gif', 'bmp', 'tif', 'webp', // image files
      'odt', 'epub', 'docx', 'pdf' // document files
    ];

Aside from that, we also need to initialize the arrays which will store the valid MIME types for each type of file that we will be dealing with. Note that these doesn’t include the MIME types which we will be converting to even though they’re valid. This is because we’ll mainly use it to determine which converter to use based on the MIME type:

    const video_mimetypes = [
      'video/x-matroska', 'video/x-flv', 'video/quicktime', 
      'video/webm', 'video/ms-asf', 'video/x-ms-wmv', 'video/x-msvideo'
    ]; // convert to mp4
    const audio_mimetypes = [
      'audio/mp4', 'audio/m4a', 
      'audio/ogg', 'audio/vnd.wav'
    ]; // convert to mp3
    const image_mimetypes = ['image/bmp', 'image/webp', 'image/tiff']; // convert to png
    const doc_mimetypes = [
      'application/epub+zip', 
      'application/vnd.openxmlformats-officedocument.wordprocessingml.document', 
      'application/vnd.oasis.opendocument.text'
    ]; // convert to pdf

Note: There are more file types in the arrays above compared to the support table I’ve shown earlier. This is because I don’t have time to test out every file type. But the software used for converting seems to support the file types above, that’s why I included them.

Next, we initialize Multer, an Express middleware for handling the file uploads. Below we’re setting the file size limit and the number of files to handle per upload. We’re also adding file type validation via fileFilter. This is where we check for the valid_filetypes that we created earlier. Note that this one simply relies on the file extension to determine if the file is valid:

    const upload = multer({
      limits: {
        fileSize: 30 * 1024 * 1024, // 30 MB
        files: 1 // only 1 file per upload
      },
      fileFilter: (req, file, cb) => {
        if (valid_filetypes.some(ext => file.originalname.endsWith("." + ext))) {
          return cb(null, true);
        }
        return cb(new Error('Invalid file type'));
      }
    });

Since the validation above is a bit weak, we also add the function for validating the file type based on its actual data. Here, we use the file-type module to get the MIME type based on its array buffer:

    const validateFileType = async (req, res, next) => {
      try {
        const mime = filetype(req.file.buffer);
        if (!mime || !valid_filetypes.includes(mime.ext)) {
          return next(new Error('invalid file type.'));
        }
      } catch (err) {
        return next(new Error('invalid file type.'));
      }

      next();
    }

Next, set the uploads folder inside the server as the directory for the static files. This is where all the uploaded files will go. The converted files will also live in here (the original file will be deleted):

    app.use(cors());
    app.use('/uploads', express.static('uploads')); // add this

The last thing that we need to do is to add the route for handling file uploads. As you’ve seen in the code for the app earlier, this is where the files are submitted for processing. Here, we make use of Multer as a middleware for this specific route. We can do that by passing upload.single(``'``fileData``'``) as the second argument to the post() method. This is Multer’s method for handling single file upload. The third argument is the function for validating the file type that we created earlier. The fourth argument is the callback function that gets executed if the validation passes:

    app.post('/upload', upload.single('fileData'), validateFileType, (req, res, next) => {

       // next: add code for processing the request
    });

Inside the callback function for processing the request, we know that the file is a valid file so we can go ahead and use the fs module to save it to the uploads folder:

    const filename = randomstring.generate(); // generate random file name
    fs.writeFile(`uploads/${filename}`, req.file.buffer, (err) => {

      if (err) {
        console.log("error: ", err);
        res.status(400).send(new Error('error getting the file'));
      }

      // next: add code for converting the file
    });

Next, we add the code for converting the file to their desired format. In this case, we need to use a different software for converting based on the type of file:

Thankfully, each software already has its own Node library which we can use to interact with it. We’ve already installed those earlier on the setup section. Go ahead and add the code for determining the type of file. This uses the array of MIME types we created earlier to determine which converter to use:

    const mime_type = req.file.mimetype; // get the file's mime type
    const file_path = `uploads/${filename}`; // the file upload path
    const file_ext = mimetypes(mime_type); // get the file extension based on the mime type

    const host = req.get('host'); // your current host (e.g xxxxyz.ngrok.io)

    if (video_mimetypes.includes(mime_type) || audio_mimetypes.includes(mime_type)) {
      // next: add code for converting videos
    } else if (image_mimetypes.includes(mime_type)) {
      // next: add code for converting images
    } else if (doc_mimetypes.includes(mime_type)) {
      // next: add code for converting docs
    }

For video and audio files, we make use of the ffmpeg library. This accepts the file path as its argument. You can specify the path to the output file via the output() method. Just like most operations in Node.js, converting a file is an asynchronous operation. That’s why we need to listen for the end event to get triggered. This is where we can return a response which contains the URL and the MIME type of the converted file. You’ll see that the same pattern used on all the other file conversions that we perform:

    const is_video = video_mimetypes.includes(mime_type);
    const converted_file_ext = is_video ? 'mp4' : 'mp3';

    let file = ffmpeg(file_path);
    if (is_video) {
      file = file.size('360x?'); // resize the video to 360p
    }

    file.output(`${file_path}.${converted_file_ext}`)
    .on('end', () => {
      return res.send({
        url: `https://${host}/${file_path}.${converted_file_ext}`,
        type: is_video ? 'video/mp4' : 'audio/mpeg'
      });
    })
    .run();

Next, we use the gm library to resize and convert the image files to PNG format. This makes use of ImageMagick underneath:

    gm(file_path)
      .resize(400) // resize the width, height will be resized to maintain aspect ratio
      .write(`${file_path}.png`, (err) => {
        if (!err) {
          return res.send({
            url: `https://${host}/${file_path}.png`,
            type: 'image/png'
          });
        }
      });

Lastly, we convert the document files using the node-pandoc library. The API of this library is a bit different because how you run it looks similar to how you would do it in the command line:

  • -f - the input format. In this case, we’re simply using the file extension of the source file.
  • -t - the output format. In this case, we’re specifying html5 as it’s the most versatile format. Note that the resulting file won’t actually be an HTML file. It will only be outputted in HTML, but the resulting file will still be the one that you want (specified by -o option).
  • -o - the output file. In this case, we simply append .pdf to the file path so the output is rendered on a PDF file.

The function accepts the file path as the first argument, and the command for converting the file as its second. The third argument is the callback function which will get executed once the file is converted:

    nodePandoc(file_path, `-f ${file_ext} -t html5 -o ${file_path}.pdf`, (err, result) => {
      if (!err) {
        return res.send({
          url: `https://${host}/${file_path}.pdf`,
          type: 'application/pdf'
        });
      }
    });

Lastly, if the file is either an MP4, MP3, PDF, PNG, JPEG, or GIF, we simply rename it to append the detected file extension. Earlier, we’ve simply saved the file to the filesystem without its file extension:

    fs.rename(file_path, `${file_path}.${file_ext}`, function(err) {
      if (!err) {
        return res.send({
          url: `https://${host}/${file_path}.${file_ext}`,
          type: mime_type
        });
      }
    });

Building the app

Now we’re ready to build the app. Well, it’s more like updating the existing app because most of the code has already been pre-written. As mentioned earlier, we’re only going to implement the code for uploading files to the server and using it for file hosting instead of Chatkit’s servers. Specifically, the following features are already implemented in the starter project:

  • Chat - allows the user to log in, view the chat rooms, and chat with the users in the room.
  • File attachment - allows the user to attach a file to a message. This currently makes use of Chatkit’s servers. We’ll update it so it uploads the file to our own server and simply attach the URL that points to the file when sending the message.
  • File preview - for previewing common image files (JPEG, PNG, GIF), viewing MP4 and MP3 files, and viewing PDF files.

The first thing that you need to do is update the initial state to include the default data for the uploaded file:

    // src/Chat.js
    state = {
      messages: [],
      is_picking_file: false,
      has_attachment: false,

      is_sending: false,
      is_modal_visible: false,
      video_uri: null,

      is_viewing_pdf: false,
      pdf_source: null,

      // add these:
      uploaded_media_url: null, // stores the URL pointing to the uploaded file
      uploaded_media_type: null // the mime type of the uploaded file
    };

Next, update the openFilePicker() function. Currently, it simply sets the file to be attached to the Chatkit message. This time, we want it to directly upload the file to the server. There’s a possibility that the user will pick another file, but we’re not concerned about that at this point. We’ll simply assume that the user isn’t going to change their mind.

Here’s the full code for the function:

    openFilePicker = async () => {
      await this.setState({
        is_picking_file: true
      });

      DocumentPicker.show({
        filetype: [DocumentPickerUtil.allFiles()],
      }, async (err, file) => {

        if (!err) {
          try {
            const data = new FormData();
            data.append('fileData', {
              uri : file.uri,
              type: file.type,
              name: file.fileName
            });

            const upload_instance = axios.create({
              baseURL: CHAT_SERVER,
              timeout: 20000,
              headers: {
                'Accept': 'application/json',
                'Content-Type': 'multipart/form-data',
              }
            });

            const res = await upload_instance.post('/upload', data);
            this.setState({
              uploaded_media_type: res.data.type,
              uploaded_media_url: res.data.url,
              is_picking_file: false,
              has_attachment: true
            });

          } catch (read_file_err) {
            Alert.alert("Invalid file", "File shouldn't exceed 10mb and it should be a valid video, audio, document, or image file.");
            this.setState({
              is_picking_file: false,
              has_attachment: false
            });
          }
        } else {
          this.setState({
            is_picking_file: false,
            has_attachment: false
          });
        }
      });
    }

Breaking down the code above, we first construct the form data. As you might already know some Web APIs have made it to React Native. FormData is one of them. This allows us to easily construct the multipart form data that’s necessary for handling file uploads. It’s a good thing this plays well with the React Native Document Picker. So all we have to do is supply the uri, type, and name of the file. They will be all under the fileData field:

    const data = new FormData();
    data.append('fileData', {
      uri : file.uri,
      type: file.type,
      name: file.fileName
    });

Next, we create a new axios instance for making the request. This allows us to specify the request headers and other request options:

    const upload_instance = axios.create({
      baseURL: SERVER_URL,
      timeout: 20000, // 20 seconds timeout
      headers: {
        'Accept': 'application/json',
        'Content-Type': 'multipart/form-data',
      }
    });

Next, we make the request to the server. The route where we made the request to returns the MIME type and the URL pointing to the converted file. We temporarily set this on the state until such time that the user send their message:

    const res = await upload_instance.post('/upload', data);
    this.setState({
      uploaded_media_type: res.data.type,
      uploaded_media_url: res.data.url
    });

Lastly, update the onSend() method to extract the data of the uploaded file from the state and add it as a message part:

    onSend = async ([message]) => {
      const { uploaded_media_type, uploaded_media_url } = this.state; // add this
      const message_parts = [
        { type: "text/plain", content: message.text },
        { type: uploaded_media_type, url: uploaded_media_url } // add this
      ];

      try {
        await this.currentUser.sendMultipartMessage({
          roomId: this.room_id,
          parts: message_parts
        });

        // add this
        this.setState({
          uploaded_media_type: null,
          uploaded_media_url: null,
          has_attachment: false
        });
      } catch (send_msg_err) {
        console.log("error sending message: ", send_msg_err);
      }
    }

Running the app

At this point, you’re now ready to run the app. Start by running the server and exposing it via ngrok:

    node index.js
    ./ngrok http 5000

Open the src/screens/Rooms.js and src/screens/Chat.js file and add your ngrok HTTP URL:

    const CHAT_SERVER = "YOUR NGROK HTTPS URL";

Next, run the app:

    react-native run-android
    react-native run-ios

On your Chatkit app instance, create users that you can use for logging in and also create a room. Then use the username to log in to the app. To test the app, upload any kind of file. It should return an error if the validation isn’t met. Otherwise, it will upload the file, convert it to the desired format, and return the URL to the converted file. When you send the message, the URL to the converted file will be attached to the message and you’ll be able to preview it.

Conclusion

In this tutorial, you learned how to create your very own media conversion server for Chatkit files. Specifically, you learned how to upload files from React Native to a Node.js server using the FormData API and Multer. You also learned how to make use ffmpeg, Pandoc, and ImageMagick in Node.js to convert common media files.

You can find the source code on this GitHub repo.

Clone the project repository
  • Node.js
  • React Native
  • Chat
  • JavaScript
  • Chatkit

Products

  • Channels
  • Chatkit
  • Beams

© 2019 Pusher Ltd. All rights reserved.

Pusher Limited is a company registered in England and Wales (No. 07489873) whose registered office is at 160 Old Street, London, EC1V 9BW.