ChatGPT解决这个技术问题 Extra ChatGPT

Remove ✅, 🔥, ✈ , ♛ and other such emojis/images/signs from Java strings

I have some strings with all kinds of different emojis/images/signs in them.

Not all the strings are in English -- some of them are in other non-Latin languages, for example:

▓ railway??
→ Cats and dogs
I'm on 🔥
Apples ⚛ 
✅ Vi sign
♛ I'm the king ♛ 
Corée ♦ du Nord ☁  (French)
 gjør at både ◄╗ (Norwegian)
Star me ★
Star ⭐ once more
早上好 ♛ (Chinese)
Καλημέρα ✂ (Greek)
another ✓ sign ✓
добрай раніцы ✪ (Belarus)
◄ शुभ प्रभात ◄ (Hindi)
✪ ✰ ❈ ❧ Let's get together ★. We shall meet at 12/10/2018 10:00 AM at Tony's.❉

...and many more of these.

I would like to get rid of all these signs/images and to keep only the letters (and punctuation) in the different languages.

I tried to clean the signs using the EmojiParser library:

String withoutEmojis = EmojiParser.removeAllEmojis(input);

The problem is that EmojiParser is not able to remove the majority of the signs. The ♦ sign is the only one I found till now that it removed. Other signs such as ✪ ❉ ★ ✰ ❈ ❧ ✂ ❋ ⓡ ✿ ♛ 🔥 are not removed.

Is there a way to remove all these signs from the input strings and keeping only the letters and punctuation in the different languages?

what you want to keep?
Two problems: What is EmojiParser? Doesn't seem to be part of a standard library, so this mention is not very helpful. And what characters exactly do you want to filter? You say "many more of this kind", but there are many character groups and families. We need to know more about your criteria.
IDK what your motivations behind this are, but if it's too filter text input: don't. I'm tired of being forced to use a-zA-Z. Let me write in my native language, or emojis, or whatever I want. Do I really want me calendar appointment to be called "🤦🏻‍♂️"? Yes, yes I do. Now get out of my way.
Please clarify what exactly you want to keep and remove. On the surface the question appears to be clear but because of the complexity of Unicode it is not and because of that it's impossible to provide a good answer.
this seems like a strange thing to want to do when it destroys the meaning of at least one of your examples?

N
Nick Bull

Instead of blacklisting some elements, how about creating a whitelist of the characters you do wish to keep? This way you don't need to worry about every new emoji being added.

String characterFilter = "[^\\p{L}\\p{M}\\p{N}\\p{P}\\p{Z}\\p{Cf}\\p{Cs}\\s]";
String emotionless = aString.replaceAll(characterFilter,"");

So:

[\\p{L}\\p{M}\\p{N}\\p{P}\\p{Z}\\p{Cf}\\p{Cs}\\s] is a range representing all numeric (\\p{N}), letter (\\p{L}), mark (\\p{M}), punctuation (\\p{P}), whitespace/separator (\\p{Z}), other formatting (\\p{Cf}) and other characters above U+FFFF in Unicode (\\p{Cs}), and newline (\\s) characters. \\p{L} specifically includes the characters from other alphabets such as Cyrillic, Latin, Kanji, etc.

The ^ in the regex character set negates the match.

Example:

String str = "hello world _# 皆さん、こんにちは! 私はジョンと申します。🔥";
System.out.print(str.replaceAll("[^\\p{L}\\p{M}\\p{N}\\p{P}\\p{Z}\\p{Cf}\\p{Cs}\\s]",""));
// Output:
//   "hello world _# 皆さん、こんにちは! 私はジョンと申します。"

If you need more information, check out the Java documentation for regexes.


The obvious gap between ASCII alphanumeric characters and emoji is accentated and non-latin letters. Without the OP's input on these we don't know whether this is a good answer (not my DV though)
Yeah I'm curious as to why this would possibly get downvoted. The second I saw this question, a regular expression was the absolute first thing that came to mind (P.S. since he's looking for standard characters and punctuation, I'd use something like [^\w\^\-\[\]\.!@#$%&*\(\)/+'":;~?,] but that's just me being robust and trying collect all typical characters that aren't symbols). Upvoted because this is definitely a potential solution. If he wants to add some other language characters, he can add them to the expression as necessary.
@Chris great punctuation regex example, looks extensive enough to me for some cases. Also maybe people aren't reading the whole answer then - as stated at the bottom of the answer, p{L} handles non-English alphabetical characters. I hope it's understood that I can't list extensively through every non-English alphabet in my answer as that would be impractically verbose.
This. Please and thank you. Don't try to forbid characters that cause you problems; decide what characters you allow and encode that. Then your code has a clearly defined set of test cases.
I suggest "[^\\p{L}\\p{M}\\p{N}\\p{P}\\p{Z}\\p{Cf}\\s]". This allows the general categories Letter, Mark, Number, Punctuation, Separator, and "Other, Format", as well as whitespace characters such as tab and newline.
D
Daniel Wagner

I'm not super into Java, so I won't try to write example code inline, but the way I would do this is to check what Unicode calls "the general category" of each character. There are a couple letter and punctuation categories.

You can use Character.getType to find the general category of a given character. You should probably retain those characters that fall in these general categories:

COMBINING_SPACING_MARK
CONNECTOR_PUNCTUATION
CURRENCY_SYMBOL
DASH_PUNCTUATION
DECIMAL_DIGIT_NUMBER
ENCLOSING_MARK
END_PUNCTUATION
FINAL_QUOTE_PUNCTUATION
FORMAT
INITIAL_QUOTE_PUNCTUATION
LETTER_NUMBER
LINE_SEPARATOR
LOWERCASE_LETTER
MATH_SYMBOL
MODIFIER_LETTER
MODIFIER_SYMBOL
NON_SPACING_MARK
OTHER_LETTER
OTHER_NUMBER
OTHER_PUNCTUATION
PARAGRAPH_SEPARATOR
SPACE_SEPARATOR
START_PUNCTUATION
TITLECASE_LETTER
UPPERCASE_LETTER

(All of the characters you listed as specifically wanting to remove have general category OTHER_SYMBOL, which I did not include in the above category whitelist.)


FORMAT (Cf) should be preserved also; this includes the clustering and directional overrides, without which it is impossible to write certain (unusual, admittedly) words in some languages.
@zwol Thanks for the details! I'll add it to the list.
This is the future-proof answer. Regardless of future updates to the Unicode standard, including/excluding characters based on their categories means that individual parsing of characters and the maintenance of a list is unnecessary. Of course, cursory testing of text in different languages (e.g. Chinese, Arabic etc.) should be done to ensure that the filtered categories match the text required to be allowed in the target environment.
Oh, another gotcha I should have thought of yesterday: TAB, CR, and LF are all general category Cc (Java's CONTROL). Those need to be specially whitelisted, since you almost certainly don't want to allow most of the legacy control characters.
@CJBS The problem with this approach is that it has only been partially implemented in Java. For example, Character.getType() won't tell you whether your char (or int code point since the method is overloaded) is, say, an emoticon, or a musical symbol, or an emoji character, etc. If you have a simple use case it might be fine to go down this path - it's certainly an elegant approach that is easy to comprehend - but be aware that it might break if requirements change.
K
Karol Dowbecki

Based on Full Emoji List, v11.0 you have 1644 different Unicode code points to remove. For example is on this list as U+2705.

Having the full list of emojis you need to filter them out using code points. Iterating over single char or byte won't work as single code point can span multiple bytes. Because Java uses UTF-16 emojis will usually take two chars.

String input = "ab✅cd";
for (int i = 0; i < input.length();) {
  int cp = input.codePointAt(i);
  // filter out if matches
  i += Character.charCount(cp); 
}

Mapping from Unicode code point U+2705 to Java int is straightforward:

int viSign = 0x2705;

or since Java supports Unicode Strings:

int viSign = "✅".codePointAt(0);

Very useful list. Interesting that something called EmojiParser with a method called removeAllEmojis fails to handle these... :-)
@Bergi: No, since input.codePointAt only looks at up to 2 characters at most which is a constant upper bound. Also (the newly added) i += Character.charCount(cp) skips over all characters that input.codePointAt inspected (minus 1 in some corner cases).
@OlivierGrégoire: String.chars() streams over characters not codepoints. There's a separate method String.codePoints() for that.
There are at least two problems here: you are using a "closed" list of emojis, so each year you have to extend it (but this probably isn't easily solvabile), and this code won't probably work correctly with codepoints sequences (see for example unicode.org/Public/emoji/11.0/emoji-zwj-sequences.txt)
This is basically the same approach as used by EmojiParser and it will soon fail for the same reason. New emojis are relatively frequently added to the Unicode character database and if you are now implementing a solution using the currently defined 1644 emojis for a negative rule set, the implementation will fail as soon as new emojis become available.
D
Daniel F

ICU4J is your friend.

UCharacter.hasBinaryProperty(UProperty.EMOJI);

Remember to keep your version of icu4j up to date and note this will only filter out official Unicode emoji, not symbol characters. Combine with filtering out other character types as desired.

More information: http://icu-project.org/apiref/icu4j/com/ibm/icu/lang/UProperty.html#EMOJI


Until Java is updated to include Emoji binary property, I guess this would be a good solution. The library needs to be updated often for the newly added codepoints, though.
t
talonmies

I gave some examples below, and thought that Latin is enough, but...

Is there a way to remove all these signs from the input string and keeping only the letters & punctuation in the different languages?

After editing, developed a new solution, using the Character.getType method, and that appears to be the best shot at this.

package zmarcos.emoji;

import java.util.Arrays;
import java.util.HashSet;
import java.util.Set;

public class TestEmoji {

    public static void main(String[] args) {
        String[] arr = {"Remove ✅, 🔥, ✈ , ♛ and other such signs from Java string",
            "→ Cats and dogs",
            "I'm on 🔥",
            "Apples ⚛ ",
            "✅ Vi sign",
            "♛ I'm the king ♛ ",
            "Star me ★",
            "Star ⭐ once more",
            "早上好 ♛",
            "Καλημέρα ✂"};
        System.out.println("---only letters and spaces alike---\n");
        for (String input : arr) {
            int[] filtered = input.codePoints().filter((cp) -> Character.isLetter(cp) || Character.isWhitespace(cp)).toArray();
            String result = new String(filtered, 0, filtered.length);
            System.out.println(input);
            System.out.println(result);
        }

        System.out.println("\n---unicode blocks white---\n");
        Set<Character.UnicodeBlock> whiteList = new HashSet<>();
        whiteList.add(Character.UnicodeBlock.BASIC_LATIN);
        for (String input : arr) {
            int[] filtered = input.codePoints().filter((cp) -> whiteList.contains(Character.UnicodeBlock.of(cp))).toArray();
            String result = new String(filtered, 0, filtered.length);
            System.out.println(input);
            System.out.println(result);
        }

        System.out.println("\n---unicode blocks black---\n");
        Set<Character.UnicodeBlock> blackList = new HashSet<>();        
        blackList.add(Character.UnicodeBlock.EMOTICONS);
        blackList.add(Character.UnicodeBlock.MISCELLANEOUS_TECHNICAL);
        blackList.add(Character.UnicodeBlock.MISCELLANEOUS_SYMBOLS);
        blackList.add(Character.UnicodeBlock.MISCELLANEOUS_SYMBOLS_AND_ARROWS);
        blackList.add(Character.UnicodeBlock.MISCELLANEOUS_SYMBOLS_AND_PICTOGRAPHS);
        blackList.add(Character.UnicodeBlock.ALCHEMICAL_SYMBOLS);
        blackList.add(Character.UnicodeBlock.TRANSPORT_AND_MAP_SYMBOLS);
        blackList.add(Character.UnicodeBlock.GEOMETRIC_SHAPES);
        blackList.add(Character.UnicodeBlock.DINGBATS);
        for (String input : arr) {
            int[] filtered = input.codePoints().filter((cp) -> !blackList.contains(Character.UnicodeBlock.of(cp))).toArray();
            String result = new String(filtered, 0, filtered.length);
            System.out.println(input);
            System.out.println(result);
        }
        System.out.println("\n---category---\n");
        int[] category = {Character.COMBINING_SPACING_MARK, Character.COMBINING_SPACING_MARK, Character.CONNECTOR_PUNCTUATION, /*Character.CONTROL,*/ Character.CURRENCY_SYMBOL,
            Character.DASH_PUNCTUATION, Character.DECIMAL_DIGIT_NUMBER, Character.ENCLOSING_MARK, Character.END_PUNCTUATION, Character.FINAL_QUOTE_PUNCTUATION,
            /*Character.FORMAT,*/ Character.INITIAL_QUOTE_PUNCTUATION, Character.LETTER_NUMBER, Character.LINE_SEPARATOR, Character.LOWERCASE_LETTER,
            /*Character.MATH_SYMBOL,*/ Character.MODIFIER_LETTER, /*Character.MODIFIER_SYMBOL,*/ Character.NON_SPACING_MARK, Character.OTHER_LETTER, Character.OTHER_NUMBER,
            Character.OTHER_PUNCTUATION, /*Character.OTHER_SYMBOL,*/ Character.PARAGRAPH_SEPARATOR, /*Character.PRIVATE_USE,*/
            Character.SPACE_SEPARATOR, Character.START_PUNCTUATION, /*Character.SURROGATE,*/ Character.TITLECASE_LETTER, /*Character.UNASSIGNED,*/ Character.UPPERCASE_LETTER};
        Arrays.sort(category);
        for (String input : arr) {
            int[] filtered = input.codePoints().filter((cp) -> Arrays.binarySearch(category, Character.getType(cp)) >= 0).toArray();
            String result = new String(filtered, 0, filtered.length);
            System.out.println(input);
            System.out.println(result);
        }
    }

}

Output:

---only letters and spaces alike---

Remove ✅, 🔥, ✈ , ♛ and other such signs from Java string
Remove      and other such signs from Java string
→ Cats and dogs
 Cats and dogs
I'm on 🔥
Im on 
Apples ⚛ 
Apples  
✅ Vi sign
 Vi sign
♛ I'm the king ♛ 
 Im the king  
Star me ★
Star me 
Star ⭐ once more
Star  once more
早上好 ♛
早上好 
Καλημέρα ✂
Καλημέρα 

---unicode blocks white---

Remove ✅, 🔥, ✈ , ♛ and other such signs from Java string
Remove , ,  ,  and other such signs from Java string
→ Cats and dogs
 Cats and dogs
I'm on 🔥
I'm on 
Apples ⚛ 
Apples  
✅ Vi sign
 Vi sign
♛ I'm the king ♛ 
 I'm the king  
Star me ★
Star me 
Star ⭐ once more
Star  once more
早上好 ♛

Καλημέρα ✂


---unicode blocks black---

Remove ✅, 🔥, ✈ , ♛ and other such signs from Java string
Remove , ,  ,  and other such signs from Java string
→ Cats and dogs
→ Cats and dogs
I'm on 🔥
I'm on 
Apples ⚛ 
Apples  
✅ Vi sign
 Vi sign
♛ I'm the king ♛ 
 I'm the king  
Star me ★
Star me 
Star ⭐ once more
Star  once more
早上好 ♛
早上好 
Καλημέρα ✂
Καλημέρα 

---category---

Remove ✅, 🔥, ✈ , ♛ and other such signs from Java string
Remove , ,  ,  and other such signs from Java string
→ Cats and dogs
 Cats and dogs
I'm on 🔥
I'm on 
Apples ⚛ 
Apples  
✅ Vi sign
 Vi sign
♛ I'm the king ♛ 
 I'm the king  
Star me ★
Star me 
Star ⭐ once more
Star  once more
早上好 ♛
早上好 
Καλημέρα ✂
Καλημέρα 

The code works by streaming the String to code-points. Then using lambdas to filter characters into a int array, then we convert the array to String.

The letters and spaces are using using the Character methods to filter, not good with punctuation. Failed attempt.

The unicode blocks white filter using the unicode blocks the programmer specifies as allowed. Failed attempt.

The unicode blocks black filter using the unicode blocks the programmer specifies as not allowed. Failed attempt.

The category filter using the static method Character.getType. The programmer can define in the category array what types are allowed. WORKS😨😱😰😲😀.


import java.lang.Character.UnicodeBlock;, then Character.UnicodeBlock -> UnicodeBlock.
All your ways failed the tests.
@Oleg no, look again, the white list example.
Something must be wrong with my eyes or my monitor, I can't see is 早上好 and Καλημέρα
Note that the Java language is a little slow supporting newer Unicode versions... For example Java 10 supports only Unicode 8 (so its character classes describe only Unicode 8 characters)... So many emojis aren't presente (see docs.oracle.com/javase/10/docs/api/java/lang/Character.html, Character information is based on the Unicode Standard, version 8.0.0.)
l
liheyuan

Try this project simple-emoji-4j

Compatible with Emoji 12.0 (2018.10.15)

Simple with:

EmojiUtils.removeEmoji(str)

It doesn't work for me on: "🤑 💘", the result is "🤑"
A
Adil B

Use a jQuery plugin called RM-Emoji. Here's how it works:

$('#text').remove('emoji').fast()

This is the fast mode that may miss some emojis as it uses heuristic algorithms for finding emojis in text. Use the .full() method to scan entire string and remove all emojis guaranteed.


The question was in Java, hence a jQuery plugin is not relevant here.