Breaking News: Grepper is joining You.com. Read the official announcement!

how to add special token to bert tokenizer

Clever Cardinal answered on January 21, 2021 Popularity 6/10 Helpfulness 7/10

answer how to add special token to bert tokenizer

how to add special token to bert tokenizer

Comment

special_tokens_dict = {'additional_special_tokens': ['[C1]','[C2]','[C3]','[C4]']}
num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)
model.resize_token_embeddings(len(tokenizer))

Popularity 6/10 Helpfulness 7/10 Language python

Source: Grepper

Tags: add python token

Link to this answer
Share Copy Link

Contributed on Jan 21 2021

Clever Cardinal

0 Answers Avg Quality 2/10

Continue with Google

By continuing, I agree that I have read and agree to Greppers's Terms of Service and Privacy Policy.

how to add special token to bert tokenizer

Contents

More Related Answers

how to add special token to bert tokenizer

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.